Skip to main content

ASSURESOFT INSIGHTS

The Nearshore Advantage

Software development security pipeline illustrating shift left vulnerability scanning

What Happens When Security Finds the Problem One Day Before Release?

Late security findings are expensive for reasons that have little to do with the complexity of the fix. This article examines how the timing of security feedback affects the cost of a vulnerability and why teams need to think carefully about where those checks sit in the delivery process.

The security finding arrived one day before the planned release and pointed to a library that no one on the team had added directly. It had been pulled in by another dependency, several levels removed from the application code they were working on. That detail did not make the problem any smaller: the vulnerable component was still being released, so the team postponed the release and went back through the dependency chain to find a safe way forward. 

Instead of closing out the work, engineers were back in the dependency tree trying to understand a library they had never selected directly. The delay started there. 

The case was later presented at AWS Community Day Bolivia, including the architecture involved and the subsequent changes. The full talk is available in the recorded session.

A developer could open a pull request that’s stopped by an issue that poses no immediate threat to the release. The workflow treated that finding as a stop condition, so the code could not move forward until someone reviewed or resolved it. 

Security had been concentrated near the end of the process. By the time the scan ran, several rounds of code and dependency changes had already moved through the pipeline. A library could also acquire a newly disclosed CVE after it had been added to the codebase, even if nobody had touched that dependency since. Release week was when all of that finally became visible.

When a security finding becomes a delivery problem

If an SQL injection risk is identified while a developer is still working on the code, it can usually be addressed within the same task. The developer changes the query, reruns the affected tests, and keeps going. The problem looks very different once the same issue arises after the release has already been prepared. 

If that SQL injection flaw appears after staging sign-off, someone has to tell the release owner that tomorrow’s build is no longer a sure thing. The fix may be straightforward, but the team has already finished testing that version and now has to reopen work it thought was done. 

Shift-left security addresses that timing problem by moving security checks into earlier stages of development, while changes are still comparatively cheap to make. Feedback can appear in the IDE, around commits, during pull request reviews, or through static application security testing (SAST), rather than waiting for a late-stage security review.

A warning that appears while the developer is still inside the task is easier to absorb. Once the same warning appears after staging, the team no longer changes unfinished code. It is changing a build that other people already expect to ship.

The benefit comes from reducing the amount of work that has already accumulated around the problem. A vulnerability caught during development may require a code change and another test run. The same vulnerability identified during release planning can force several teams to stop what they are doing and decide whether the release should proceed at all.

The later a vulnerability is found, the costlier its consequences can become, especially when they are that last-minute message nobody wants to read: tomorrow’s release is no longer shipping. By then, the vulnerability has already become more than a coding issue. That is the cost of getting the security signal after the rest of the organization has already started treating the work as finished. 

Security coverage has to match where risk actually enters

Timing was only one weakness exposed by the release incident. The vulnerability did not come from code written directly by the team. It entered through a transitive dependency.

The team’s own code was not the source of the problem. The vulnerable library had been pulled in as a transitive dependency, so a clean SAST result would not have exposed it. 

In this case, the AWS environment already uses controls such as WAF, VPCs, Secrets Manager, and least-privilege access. Those measures addressed infrastructure and access risks, but they could not identify the dependency that caused the release issue.

Software composition analysis, or SCA, is the check that reaches that part of the application. It inspects the third-party packages that arrive with the codebase, including libraries developers may never have selected themselves.

New vulnerabilities can also be disclosed in existing dependencies without any change to the codebase. A package that looked clean at the time of its introduction can later be associated with a newly disclosed CVE. That is why dependency scanning cannot be treated as a one-time check when the package enters the repository. Container, infrastructure, and runtime checks cover other parts of the system. Each scanner earns its place by catching something the others cannot see.

More detection creates a new decision problem

A low-severity finding could leave a pull request waiting even when it posed no immediate threat to the release. The developer had already done the work, but the code was not going anywhere until someone reviewed or resolved the alert.

The first configuration gave too many alerts, which stopped work. A minor issue could hold up a pull request even though nobody would have postponed a release over it. The pipeline eventually had to separate the act of finding something from the decision to stop work because of it.

Severity helped, but the team also considered whether the issue was exploitable, exposed, and relevant to how the component was used. A finding with a known exploit on an exposed component deserved immediate attention. Another finding could stay open without freezing unrelated work.

Noise also became part of the problem. When engineers repeatedly opened alerts that did not reflect how the software was actually being used, the scanner became easier to ignore. New rules were therefore introduced without immediately giving them blocking power. That gave the team a chance to see which alerts were useful and which ones needed adjustment before developers were forced to stop for them.

False positives need somewhere to go

A scanner may flag code that appears dangerous based on the pattern alone, even though the vulnerable path cannot actually be reached in that application. A transitive library may also contain an affected function that the product never calls.

Those findings still need review, but forcing the same argument to happen every time the scanner runs does not improve security. The team used suppressions for cases it had already investigated. Those exceptions needed a reason and an owner, and temporary ones had to come back for review rather than disappearing permanently into the configuration.

Repeated dismissals also told the team something about the rule itself. If developers kept reaching the same conclusion every time an alert appeared, the rule needed another look.

Decide what a finding can stop before it appears

A security control becomes much easier to operate when the consequence of a finding has already been decided.

Teams need to know which issues block a pull request, which ones stop deployment, which ones create follow-up work, and who owns that work once delivery continues. Without those rules, every new scanner adds another decision that has to be made after the finding appears.

Before adding another scanner, the team needs to decide what happens when it fires. If a finding is serious enough to stop a merge, that rule should already be clear. If it only requires follow-up, it should not default to a release decision. When those decisions are left until release time, security becomes another source of uncertainty. When they are built into the pipeline, the response is clearer long before a release is under pressure.

What a stronger security feedback loop looks like

A useful security finding arrives while someone can still act on it, comes from a control that can actually see the relevant risk, and triggers a response proportionate to the issue's seriousness.

Weakness in any one of those areas creates a different problem. Feedback can arrive too late, important risks can remain invisible, or the pipeline can block work that never needed to stop.

AI-assisted development will put more pressure on how teams produce and modify more code in less time. More code and more dependencies mean more findings to review, which makes indiscriminate blocking even harder to sustain.

The release incident illustrates the larger point. Security checks create value when they surface the right risk early enough for the team to act and when the pipeline already knows what to do with the result. Anything less leaves engineering making high-stakes delivery decisions at exactly the wrong moment.

Tags

Engineering Team

Engineering Team

AI-powered experts

A multidisciplinary team of AssureSoft engineers specializing in AI, data intelligence, product development, platform architecture, cloud, DevOps, enterprise and core systems, software quality, security, and UI/UX.

Drawing on their experience across complex technology projects, the team shares practical and technical insights on combining AI-driven productivity with human standards to help engineering leaders build software that delivers lasting value.