Cloud Provider Picks Native Object Store Over Third-Party Cache, Citing Latency and Lock-In

Archi3 min read

cloudstorageinfrastructuredecisions

Contents
  1. The 40-Millisecond Gap That Forced a Cloud Provider's Hand
  2. Weighing the Options
  3. The Build-Out
  4. What the Decision Means for Customers
  5. The Trade-Off Nobody Talks About

The 40-Millisecond Gap That Forced a Cloud Provider's Hand#

On a Tuesday morning in late February, the engineering leadership at Northwind Cloud faced a choice that would shape the next two years of its platform. The company, a mid-sized provider with 14 data centers across North America and Europe, had been testing a third-party caching layer to sit in front of its S3-compatible object store. The tests showed a 40-millisecond improvement in p99 read latency for small objects. But the same tests revealed a hidden cost: the cache vendor's licensing model would add 18 percent to the cost of every stored terabyte, and the integration required Northwind to route all metadata queries through the vendor's control plane.

That last detail was the sticking point. Northwind's CTO, in an internal memo reviewed for this report, wrote that routing metadata through a third party would make it impossible to migrate storage backends without a six-month re-architecture. The team had spent the previous year building a multi-tenant storage layer designed to switch between hardware generations. The cache would have undone that work.

Weighing the Options#

The decision came down to two paths. Path A: adopt the third-party cache, accept the 18 percent cost increase, and gain the 40-millisecond improvement for the roughly 12 percent of workloads that are latency-sensitive. Path B: build an in-house cache using open-source components, keep metadata in-house, and accept a longer development timeline. The team estimated six months to reach parity with the vendor's feature set, with two engineers dedicated full-time.

According to two people familiar with the discussions, the finance team initially favored Path A because it required no additional headcount. But the engineering team pushed back with a cost model that showed the vendor's licensing fees would exceed the cost of two engineers after 18 months, assuming current growth in stored data. Northwind's storage footprint had grown 60 percent year over year, and the model projected similar growth for at least two more years.

The Build-Out#

The team chose Path B. They started with an open-source caching proxy and added a custom metadata shard that kept all query routing within Northwind's control plane. The first production deployment went live in early May, covering three of the 14 data centers. By mid-June, the cache was handling 35 percent of read traffic with a p99 latency improvement of 32 milliseconds, slightly less than the vendor's 40 but without the licensing overhead.

The build-out was not without friction. The team hit a snag in April when the open-source proxy's connection pooling caused memory spikes under high concurrency. They patched it by capping connections per node and adding a queue depth monitor. That fix added two weeks to the timeline.

What the Decision Means for Customers#

For Northwind's customers, the immediate impact is a price hold on storage through the end of the year. The company had signaled in January that a storage price increase was likely in the second half. That increase is now off the table, according to a customer email sent last week. The email also noted that the in-house cache will be extended to all 14 data centers by October.

The longer-term bet is about portability. By keeping metadata in-house, Northwind can switch storage hardware vendors without renegotiating cache contracts. That flexibility has already paid off: in July, the company signed a new hardware deal with a disk manufacturer that undercut its previous supplier by 9 percent. The migration is scheduled for the first quarter of next year.

The Trade-Off Nobody Talks About#

The decision also carries risk. Building in-house means Northwind owns the maintenance burden. The two engineers dedicated to the cache project are now responsible for patching, monitoring, and scaling it. If one leaves, the project loses half its capacity. The company has not yet hired a third engineer for the team, though a job posting went up in June.

There is also the question of feature parity. The vendor's cache offered automatic invalidation across regions, a feature Northwind's in-house version does not yet have. For now, the team is handling invalidation with a manual trigger that runs every 15 minutes. That works for most workloads, but it adds a delay that some customers may notice.

For Northwind, the calculus was clear: a 40-millisecond latency gain was not worth surrendering control of the metadata layer. Whether that bet holds up as data volumes grow is the question the next year will answer.

More articles