Skip to main content
← /writing
  • #agentic-sdlc
  • #platform-engineering
  • #engineering-leadership

Black October Exposed the Remediation Bottleneck

AI is making some old flaws easier to exploit. Black October exposed the harder problem: delivering a safe fix when both delay and the newest release can put you at risk.

Vinny Carpenter14 min read2.7k words

never the stack · audio edition

Black October Exposed the Remediation Bottleneck

23:04

On October 17, a researcher known as Wayne found recoverable private keys in eight eMudhra intermediate certificates. His timeline on Mozilla's incident record puts the discovery 27 minutes after he started experimenting. Mozilla independently recovered the keys less than four hours after his first email. Every one was an integer below 256, and one was the number 4. Mozilla's Dennis Jackson then gave a few local AI agents a one-line capture-the-flag prompt, and they recovered a key on the first attempt.

That was a catastrophic failure of key generation. The harder problem afterward was getting every system that trusted those certificates to stop accepting them.

Four days later, PHP released fixes for PHP2Shell, flaws in the interpreter that could allow remote code execution and memory disclosure. Details had leaked the day before, while no patch existed, and government agencies in the Netherlands, Finland, France, and Germany took public sites offline rather than wait. On the day the fixes shipped, a poisoned LiteLLM release reached PyPI and container registries. It carried malware designed to steal credentials.

These incidents presented engineering teams with an uncomfortable combination. An urgent runtime fix needed to reach production, compromised certificate authorities needed to lose trust, and a fresh dependency release could itself be the attack.

People started calling the week Black October, and the incidents behind the name had different causes. They exposed a common operating problem: how quickly can an organization turn a finding into a safe change, then establish that the change reduced exposure everywhere it mattered?

As AI lowers the effort required to find and exploit some flaws, the capacity to deliver a safe, verified fix becomes more valuable. That capacity depends on ownership, judgment, and the platform's ability to carry a decision into production.

Faster discovery exposes slower decisions

The strongest evidence for AI's role comes from the PHP research. Wiz reports that frontier models helped its researchers adapt working exploits across applications in hours. That's a concrete account of reduced specialist effort. It doesn't prove every vulnerability is now easy to exploit, but it should prompt a fresh look at risks accepted mainly because exploitation seemed difficult.

The PHP Foundation's account explains the organizational constraint just as clearly. A core developer found the first signs of the out-of-memory problem while fixing an earlier report, and by 2024 an exploit had been theorized without a concrete attack path. Proposed fixes carried compatibility and availability tradeoffs. Work stalled amid consensus decisions and limited capacity among the developers who understood the engine. The Foundation describes a structural failure and is changing how security decisions are made.

That history matters more to me than another prediction about autonomous attackers. Discovery can make a decision urgent. It cannot supply the missing authority to make the tradeoff, the engineering capacity to implement it, or the delivery path to get it deployed.

I've seen a smaller version of that problem in platform work. At Northwestern Mutual, we reviewed why engineers kept requesting break-glass access to production. Routine tasks were using the emergency route because they had no supported pipeline path. We built those paths, as I described in Your Platform Is Teaching Your Engineers.

Emergency patching follows the same pattern. If a runtime upgrade requires someone to reconstruct how a service is built, find an owner, negotiate an exception, and test it manually, urgency doesn't remove that work. Leaders own the conditions that make a safe change repeatable.

Black October exposed a bottleneck we already had. Five changes would make that bottleneck less dangerous.

1. Know what is running and who can act

A build-time software bill of materials tells you what went into an artifact. An incident asks which affected versions are running now, whether an attacker can reach them, and who can change them. The answer needs to include the agency-hosted marketing site, the forgotten wiki, and the vendor appliance with a vulnerable runtime inside it.

Connect dependency information to deployment records and service ownership. A finding that cannot be mapped to a running service leaves the next team to repeat the investigation. A current inventory also makes targeted containment possible without treating the entire estate as equally exposed.

Decide in advance what each critical service can do when a patch is unavailable or unsafe to deploy immediately. Some can go offline. Some can serve a static page or disable a vulnerable feature. Others need a compensating control whose coverage has been checked. Name the person authorized to make that call. A 2 a.m. incident is a poor time to discover that nobody can approve taking the benefits portal offline.

Rapid7's Q3 report tracked 43 newly exploited high- and critical-severity vulnerabilities. In that cohort, exploitation followed public disclosure by an average of three days. Most advisories never get that urgent, but for a reachable, actively exploited flaw, a fixed monthly patching cycle is too slow.

2. Make the fix a safe routine

An urgent update should use a delivery path the team already trusts. Rebuild the affected artifact, test the behavior that matters, deploy progressively, and verify the running version. Decide how to contain the service if the update fails, including whether rolling back would restore the vulnerability.

I run a small version of this on my own repositories. A scheduled task on my Mac mini checks my GitHub repositories at 4 a.m., merges Dependabot and Renovate pull requests when CI is green, and attempts repairs when it fails. Human-authored pull requests get a report rather than an automatic merge.

Two personal agents cover the rest of the day. PRBot, a Grok bot I use for pull request reviews, summarizes my open pull requests and what's blocking each one. Kepler, my OpenAI Dots personal agent, triages security alerts. In the evening, my agents also check for vulnerabilities and updates and prepare changes for my review.

That routine removes work I'd otherwise have to remember. This week also exposed the limits of making a green CI result the deciding condition. Tests can pass while the update introduces malicious code or leaves the affected deployment unchanged.

I've added Renovate's minimumReleaseAge setting to my public repositories and adjusted Dependabot's cooldown configuration. The policy those tools need to enforce has two parts. Routine updates wait, so newly published versions have time for problems to surface before the normal source, compatibility, and release checks run. A verified security fix can move sooner. Confirm the affected dependency, the advisory's authenticity, and the right fixed version, then expedite the change without dropping the checks needed to trust and deploy it.

On GitHub.com, Dependabot defaults to a three-day cooldown for version-update pull requests. Security-update pull requests are exempt from that delay. Renovate's release-age gate works the same way, and security updates bypass it by default. The required checks still need to be configured correctly.

The two numbers line up uncomfortably. In Rapid7's cohort, exploitation followed disclosure by three days on average, and Dependabot now holds routine version updates for three days. A security fix that travels the routine lane can arrive about when exploitation starts. That's why the platform has to keep routine soak time separate from verified security remediation.

A three-day delay would have outlasted LiteLLM's brief PyPI exposure in a routine update path that respected it. It wouldn't have protected a build that installed the latest version directly. Every automatic update path needs to honor the policy, including the agent that can merge the pull request.

Three panels showing personal agents watching my repositories: PRBot scanning four open draft PRs, PRBot following up on review and CI status the next morning, and Kepler investigating a Dependabot Zod security alert alongside a Stagehand compatibility constraint.

Examples from this week's agent reports across PRBot and Kepler. My 4 a.m. routine merges only bot pull requests on green CI after cooldown; everything requiring judgment comes to me.

The test environment also needs protection. The LiteLLM payload ran at Python startup, so a dependency test could become the victim before any merge. Run those checks in isolated jobs without release credentials or unrelated secrets.

The judgment inside a dependency update

The third panel above, Kepler's investigation of a GSD alert, shows why the workflow needs project context. The follow-up exchange below shows how it resolved. When Kepler investigated the Dependabot warning about the MCP server's Zod dependency, it surfaced an existing constraint. An earlier upgrade had broken Stagehand's schema type-checking, so the project had deliberately held the dependency at 4.4.3.

The proposed response was a targeted update to the MCP package, with lockfile changes and checks for oversized inputs, MCP functionality, and type-checking. A blanket dependency refresh would have ignored the reason for the existing pin.

Kepler's GSD dependency investigation identifies a Stagehand compatibility constraint, recommends a targeted MCP update, and later reports release and deployment checks while leaving Dependabot alert closure unverified.

Kepler's investigation of a GSD dependency alert. The final message reports MCP 1.3.2 published and deployment checks passing; Dependabot's alert-closure status remains unverified in the exchange.

The exchange doesn't show whether every copy of Zod was fixed or how the alert maps to an advisory. The useful work was identifying which dependency could change without breaking another part of the application. That's the judgment I want the platform to preserve. A passing build, a published package, a running deployment, and a closed alert each establish something different. The workflow should make those distinctions visible.

3. Treat trust as a dependency

The eMudhra case makes the backend deployment problem easy to miss. Browsers pushed remote blocklist updates within hours, while Mozilla's incident record flagged concern about systems without effective revocation updates. Backend services rarely have an equivalent push mechanism, and updating a browser doesn't prove that a service's certificate verifier rejects the same compromised chain.

Trust can come from an operating system store, a language package, a Java keystore, a container base image, or an application-specific bundle. Each needs an owner and an update path. Inventory the source of trust used by the running process, including overrides that differ from the platform default.

Go illustrates why the details matter. Its security advisory says most platforms use system root certificates. Go also added a verifier denylist for narrower cases that could remain affected. Applications need rebuilding to inherit that code change; updating the installed toolchain doesn't change an existing executable.

Prefer supported, centrally maintained trust sources where they fit the application. If internal services use a private certificate authority, configure their clients to trust only the intended issuers. Issuing a private certificate doesn't by itself stop a client from accepting public chains.

Automate certificate replacement, including early replacement after a compromise. Then test the result from the runtimes you operate. A controlled check that rejects a known distrusted chain provides stronger evidence than a release note saying a bundle was updated.

4. Protect the authority to change the release process

LiteLLM's October 23 investigation replaced its initial account of the compromise. An Agent Platform master key had been exposed publicly in May and never revoked. The attacker used it to obtain an organization owner's GitHub token, then stole additional maintainer tokens through CI workflows. Owner access let the attacker override branch protection and change the release source to a lookalike repository. Another stolen token approved publication.

The leadership question is who can change the conditions under which a release is approved. Two approvals offer limited separation if one compromised environment can obtain both identities. Branch protection offers limited protection if the compromised identity can disable it.

Give maintenance workflows their own identities, restrict their permissions, and keep release administration outside their ordinary reach. Revoke exposed credentials, even after the visible copy has been removed. Preserve audit evidence outside the systems whose owners can erase the operational history.

Use workload identities where supported. A gateway calling Amazon Bedrock through an IAM role can avoid storing a long-lived access key for that service. Temporary credentials still need narrow permissions, and a build job should receive model access only when its work requires it.

Verify artifact provenance against the repository, builder, and build inputs you expect, as SLSA's verification guidance describes. A valid signature identifies a signing process; it doesn't prove the process built appropriate code. Pin deployment artifacts by digest and send changes to those digests through the same review path.

This extends the argument in The Agent Is Wearing Your Badge and The Prompt Is Not the Permission. Automation inherits the authority we give it, including any authority to weaken its own safeguards.

The incidents also connected directly. LiteLLM's investigation acknowledges that the team has been slow to ship its eMudhra trust-store fix, and it says it won't release until it's confident the release is safe. A team that cannot trust its publishing path has lost part of its ability to remediate the next problem.

5. Fund the work after the finding

Use model-assisted security review on your own code, particularly components that process untrusted input before authentication. Parsers, upload handlers, template engines, and memory-unsafe components are useful places to start. Pair that work with people who can reproduce the finding, assess reachability, choose a response, and verify the repair.

The PHP Foundation reports that its team handled more than 100 invalid, duplicate, or unreproducible reports while publishing 15 CVEs since July. More reports create work even when they don't identify more vulnerabilities. Models can help with that triage by reproducing reports, grouping duplicates, and drafting a first assessment, while people decide the response. A discovery program still needs a triage and remediation budget alongside its scanning budget.

That obligation extends to the open source your business depends on. Fund maintainers and contribute engineering time for diagnosis, compatibility testing, and supported fixes. Purchasing another way to generate findings does little for a project that lacks people able to resolve them.

Structural improvements belong in that investment too. Reducing what exposed components can access, isolating risky parsers, and selectively replacing memory-unsafe code can reduce future remediation demand. Those changes still need clear scope and validation. AI assistance doesn't remove the compatibility work that made them difficult in the first place.

Measure the path to verified closure

The measurement should follow the service through the response. Track the time needed to establish exposure, apply effective containment, and deploy and verify the fix across all affected instances. Record those milestones separately. A mitigation can reduce immediate risk while the permanent fix remains open.

Keep the remaining exposure visible. A median can improve while one critical service stays vulnerable for weeks. Track the oldest unresolved high-risk exposure and the instances still awaiting remediation, along with the failures and rollbacks caused by urgent changes.

These measure your response. A vulnerability may have been exploitable long before the advisory, so they undercount total exposure. They do tell you which part of the response your organization can improve.

I'm less certain about how quickly the balance between discovery and repair will change. Research demonstrations don't tell us how much faster every attacker will move, and defenders can use the same tools. I'd revise my urgency if organizations absorbed more actionable findings while reducing unresolved exposure, keeping change failures stable, and preserving their release safeguards.

I'd be more concerned if expedited updates became the default and teams routinely bypassed the checks meant to make those updates safe. That would mean the organization had accelerated changes without increasing its capacity to evaluate them.

Pick one service and trace its last relevant security advisory through that process. Where did the time go: identifying the owner, choosing a compatible fix, waiting for permission, rebuilding an artifact, or proving the final instance was updated?

The next platform investment should shorten that specific delay while preserving the evidence needed to trust the result.

// found this useful? share it

Post on X Share to LinkedIn
Vinny Carpenter

Written by Vinny Carpenter

VP Engineering · 30+ years building software

I lead engineering teams building cloud-native platforms at a Fortune 100 company. I write about engineering leadership, AI-assisted development, platform strategy, and the hard lessons that come from shipping at scale.

keep reading