Cloud CPU Credits: Who Feels the Throttle First
Archi4 min read
Contents
Field notes from a week of CPU credit tests#
We rented six cloud instances across three providers and one private OpenStack lab. Four were burstable: 2 vCPU, 4 GB RAM, billed by the hour with a CPU credit balance. Two were dedicated: same core count, no credit system. Each ran the same small API, a Redis cache, and a Postgres sidecar. Load came from k6 scripts: a steady 30 requests per second, then three bursts to 180 rps for ten minutes, then a nightly backup job that compressed 12 GB.
The first number that mattered was not average CPU. It was p95 latency. On the quiet burstable boxes, the API answered in 35 to 45 ms. Once credits fell below 20 percent, p95 crossed 300 ms. One box hit 1.2 seconds. Dedicated instances moved from 38 ms to 61 ms under the same burst.
The credit balance is a second bill#
A burstable instance does not hide its billing logic. It sells a baseline and a bucket of credits. Run above baseline and the bucket drains. The invoice may still look normal. The user experience does not. We watched an endpoint that was fine at 09:00 become slow at 11:00 because a 07:00 report job had spent the morning's credits.
This lands hardest on teams with one person on call. A solo developer running a small SaaS, a two-person agency hosting client sites, a university lab running a public dataset: they chose burstable because the price was clear for idle hours. They do not have a spare cluster to absorb a report job.
Spiky work is the trap#
The least affected workload was steady. A low-traffic internal tool at 4 percent CPU never touched the credit floor. The worst was spiky and predictable: a nightly ETL job, a Monday morning dashboard refresh, a weekly PDF batch. Those jobs are invisible in monthly averages. They arrive, take the bucket, and leave a slow afternoon behind.
We tested this directly. On burstable test B, we moved the backup from 02:00 to 13:00. The p95 for interactive traffic rose from 44 ms to 260 ms within 23 minutes. Moving the job to a dedicated worker kept the API near 50 ms and cost 14 dollars more per month at the listed on-demand rate. That is the trade small teams face: a clean invoice or a clean afternoon.
CPU steal matters more than CPU usage#
Inside a shared host, the hypervisor can hand cycles elsewhere. Our monitoring caught cpu steal spikes from 1 percent to 22 percent on two burstable instances. The app's own CPU graph looked calm. The request queue did not. Teams that only watch CPU utilization will miss this. They will add application workers, which makes the queue worse because more processes compete for the same throttled core.
The fix is boring. Track steal, credit balance, and p95 by endpoint on the same dashboard. Set an alert when credits fall below 30 percent, not when the instance is already slow. For batch jobs, use a separate instance class or a scheduled container that stops when the job ends.
Who pays the most#
The billing model hits three groups hardest. First, small teams without a platform engineer. They read the price page, not the credit math. Second, public-interest projects with seasonal traffic. A voter lookup site or a food bank locator can sit idle for weeks, then take a 50x spike on one news day. Third, anyone running staging and production on the same account. A load test in staging can drain credits that production needs.
The group least affected is the one with a dedicated pool and 30 percent headroom. That is not a moral failing. It is a budget line. A 2 vCPU dedicated instance often costs two to three times the burstable price. A small team can pay it, but then it loses the buffer for a second region or a backup.
Field notes for the next migration#
We ended the test with a simple rule: put interactive traffic and batch traffic on separate billing units. If that means two small instances instead of one larger one, fine. The latency graph will show the difference before the invoice does. Check credit balance after every deploy, not once a quarter. The deploy that adds a cron job is the deploy that changes the CPU profile.
One more number stuck with us. On the burstable instance with the lowest credit balance, 8 percent of requests timed out at 5 seconds during the largest burst. On the dedicated instance, 0.1 percent did. The price gap was 22 dollars. For a hobby project, that gap is real. For a service with paying users, it is often the cheapest reliability work on the list.
More articles