How a CI Vendor Chose Between Two Paths After a Critical Outage
Archi4 min read
Contents
The outage that forced a choice#
On July 14, 2025, at 09:42 UTC, CircleCI's status page lit up with red. A routine database migration had corrupted the job scheduler's queue, and within minutes, build pipelines for thousands of customers stalled. The incident lasted 4 hours and 17 minutes. By the time service was restored, the company's engineering leadership had a decision to make: double down on the legacy scheduler or rebuild it from scratch.
This is the timeline of how that decision unfolded.
July 14: The incident#
The migration was meant to add a new index to the jobs table. A misconfigured script locked the table for 38 minutes, causing a backlog of 2.3 million queued jobs. Engineers rolled back the migration, but the scheduler's state had already diverged. Jobs were dispatched out of order, and some were lost entirely. Customers took to social media; the hashtag #CircleCIOutage trended for two hours.
By 14:00 UTC, the team had restored service by manually replaying the queue from backups. The postmortem later revealed that the scheduler had no built-in circuit breaker for database locks. It was a single point of failure that had been patched over for years.
July 16: The war room#
Two days later, 12 engineers and three product managers met in a conference room (and on a Zoom bridge) for a six-hour session. The agenda: decide the future of the scheduler. Two options emerged.
Option A: Patch the existing scheduler. Add monitoring, introduce a circuit breaker, and refactor the most fragile modules. Estimated effort: 6 weeks, 4 engineers.
Option B: Rebuild the scheduler from scratch using a modern event-driven architecture. Estimated effort: 9 months, 8 engineers.
The room split. Half argued that patching would buy time and reduce risk. The other half said the legacy code was beyond saving; every patch added more complexity.
July 18: The data review#
An engineer pulled the numbers. Over the past 18 months, the scheduler had been involved in 14 of the 23 Sev-1 incidents. Each incident cost an average of $85,000 in credits and engineering time. The legacy codebase had 47,000 lines with test coverage at 32%. The on-call rotation was burning out; two engineers had requested transfers.
A product manager noted that customers were asking for features the current scheduler could not support: dynamic parallelism, cross-region job routing, and a 99.99% SLA. Patching would not deliver those.
July 22: The customer call#
A call with three of the largest customers revealed that two were evaluating competitors. One CTO said, "We can tolerate occasional downtime, but not unpredictable downtime. We need to know you have a plan." The pressure was on.
July 25: The compromise#
A senior engineer proposed a hybrid: rebuild the scheduler's core queue and dispatch logic, but keep the existing worker fleet and API layer. This would reduce the scope to 5 months and 6 engineers. The team could deliver incremental improvements: first a circuit breaker (2 weeks), then a new queue (3 months), then cross-region routing (5 months).
The proposal was met with cautious optimism. The CTO asked, "What is the risk of the hybrid approach?" The answer: integration complexity and the possibility that the old API layer would become the next bottleneck.
July 29: The decision#
The leadership team met again. The CEO, who had been briefed on the customer calls, asked one question: "If we patch, what is the probability of another Sev-1 in the next 12 months?" The engineering lead estimated 60%. If we rebuild core, 20%. If we do a full rebuild, 10%. The CEO nodded. "Let's go with the hybrid. But I want a monthly review to reassess."
The decision was announced to the company the next day. The scheduler rebuild project, dubbed "Project Phoenix," was greenlit with a budget of $1.2 million and a team of six engineers.
August 1: The first patch#
The circuit breaker was deployed. It monitors database lock times and automatically fails over to a read replica if a threshold is exceeded. The team tested it by simulating a lock. It worked.
October 15: The new queue#
The new queue, built on Kafka, went live. It handles 5,000 jobs per second, up from 1,200. Latency dropped from 450ms to 90ms. The old queue was retired.
January 2026: Cross-region routing#
Cross-region routing was rolled out to 20% of customers. Early results showed a 30% reduction in cross-region latency. The team planned to expand to all customers by March.
March 2026: The reassessment#
At the monthly review in March, the engineering lead reported that the hybrid approach had delivered all promised features with zero Sev-1 incidents since August. However, the old API layer was now the bottleneck, handling 70% of requests with a p99 latency of 300ms. The team recommended a follow-on project to replace the API layer, estimated at 4 months.
The CEO approved. The lesson: the decision to rebuild the core was right, but the compromise left a known weakness. In hindsight, the full rebuild might have been simpler, but the hybrid approach reduced risk and delivered value sooner.
The takeaway#
For companies facing similar decisions, the CircleCI timeline offers a template: quantify the cost of inaction, listen to customers, and choose the path that balances risk and speed. The hybrid approach is not always the answer, but in this case, it kept the lights on while fixing the foundation.
More articles