The 4 KB Setting That Doubled Our Cloud Bill (And How We Found It)

Archi4 min read

cloudstoragecosts

Contents
  1. The line item nobody could explain
  2. How the test was set up
  3. The detail: a buffer size hiding in a config file
  4. What the numbers looked like
  5. Where the defaults come from, and why they stick
  6. Three checks worth running this week
  7. The part that still bugs me

The line item nobody could explain#

The bill arrived on a Tuesday, and the storage number was wrong. Not wildly wrong. Just about double what our math said it should be. We had run the same benchmark against three object storage tiers for six weeks, logged every request, and the per-gigabyte line matched our spreadsheet to within 2 percent. The request count did not.

This is field notes from that test, because the thing that broke it was not a region, a tier, or a provider. It was a 4 KB default in a client library that nobody on the team had set on purpose.

How the test was set up#

We ran three workloads against four storage configurations: a standard object store, an infrequent-access tier, a block volume, and a local SSD cache in front. The workloads were a small-file archive (average 9 KB), a medium-file batch (average 240 KB), and a single large video file (1.1 GB) uploaded in parts.

Each run lasted 14 days. We recorded cost per operation, latency at p50 and p99, and bytes actually transferred. Two engineers, one spreadsheet, no dashboards at first. That last part matters.

The detail: a buffer size hiding in a config file#

The client library we used splits uploads into chunks. The default chunk size was 8 MB. Our archive workload pushed 9 KB files. Every single one of those files triggered a full 8 MB buffer allocation on the client side, and the multipart threshold sat just low enough that roughly one in six small files crossed into multipart territory.

That meant extra requests. Not extra bytes on the wire, mostly, but extra request charges, extra metadata writes, and a p99 latency that wandered between 340 ms and 1.9 seconds for reasons that looked random until we traced them.

We changed one value: multipart threshold from 8 MB to 64 MB, chunk size from 8 MB to 1 MB. The bill for the small-file workload dropped 46 percent on the next cycle. Latency p99 fell to 410 ms and stayed there.

What the numbers looked like#

Workload Default settings Tuned settings
Small-file archive, monthly cost $1,840 $993
Small-file p99 latency 340 to 1,900 ms 410 ms steady
Medium-file batch, monthly cost $2,110 $1,970
Large video upload, monthly cost $640 $638

Notice the middle row. Tuning barely moved the medium-file batch, because 240 KB files never hit the 8 MB threshold in the first place. The large file did not care at all. The entire cost gap lived in one workload, and inside that workload it lived in one threshold value.

Where the defaults come from, and why they stick#

Client libraries pick defaults for the median case. A chunk size of 8 MB is a reasonable guess for a developer pushing images or backups. It is a terrible guess for a service that stores configuration files, log fragments, or short JSON payloads, which is most services.

The defaults are also invisible. Nobody reads the constructor options for a storage client during a migration. The value is not in the environment file, not in the infrastructure code, and not in the review. It is one line in a dependency, and it changes behavior silently.

We found ours by dumping every request the client made for a single 9 KB file. Six requests where we expected two. That was the whole investigation, once we thought to look at the request log instead of the byte count.

Three checks worth running this week#

First, log request counts by object size bucket for one day. If small objects produce more than two requests per object on average, a threshold is probably firing.

Second, set chunk size to something close to your median object size, not above it. For a 9 KB median, 1 MB is safe and small enough to avoid waste.

Third, put the multipart threshold in your infrastructure config explicitly, even if the value matches the default. The next library upgrade may change the default, and an explicit number is the only thing that survives it.

The part that still bugs me#

We ran the test to compare storage tiers. We ended up learning almost nothing about tiers and everything about a client-side threshold. The tier ranking held, roughly, but the gap between the best and worst tier was smaller than the gap between default and tuned settings within a single tier.

That is worth sitting with. Cloud cost analysis usually starts with pricing pages and reserved capacity. Our six-week bill was decided by a buffer size that no pricing page mentions, and that most teams never set. Check the small numbers first. They are cheaper to fix and they hide in plain sight.

The secondary finding, for anyone running similar tests: the local SSD cache did nothing for the small-file workload. It helped the medium batch by about 8 percent on latency and cost nothing extra. We kept it, but it was not the story.

More articles