The API Rate Limit Decision: How One Startup Chose Fairness Over Growth
Archi4 min read
Contents
The day the API started saying no#
On a Tuesday morning in March, the status page for a small developer tools company turned yellow. The company, which we will call QueryHub, provides a database query API used by about 8,000 developers. Its free tier had always been generous: 1,000 requests per minute per user. That was fine when most users were hobbyists. But over the previous six months, a handful of power users had started running production workloads on the free tier, and the shared infrastructure was buckling.
At 9:14 a.m., the first alert fired: p99 latency on the read endpoint had crossed 2 seconds. By 9:40, error rates hit 12 percent. The on-call engineer, a two-year veteran named Priya, made the call to throttle all free-tier requests to 100 per minute. The fix worked. Latency dropped. But within an hour, the support inbox filled with angry messages from developers whose apps had suddenly slowed to a crawl.
That afternoon, the founders met in a conference room with a whiteboard and a single question: how do we fix this without losing the developers who trust us?
The options on the table#
The team sketched three paths. Option A: keep the hard throttle at 100 requests per minute and tell users to upgrade to a paid plan. This was the fastest fix and would boost revenue from the 200 or so users who actually paid. Option B: build a more sophisticated rate limiter that allowed bursts, so short spikes would not trigger throttling, but sustained high usage would still be capped. This would take about two weeks of engineering time. Option C: do nothing and hope the infrastructure could be scaled up cheaply. That last one was ruled out quickly; the cloud bill had already doubled in three months.
The founders were split. The CEO, who had previously worked at a large cloud provider, argued for Option A. "We are a business," he said. "If people are using 10,000 requests per minute on a free plan, they should pay." The CTO pushed for Option B. "The users who hit the limit are our best users," she said. "They are the ones who will tell others about us. If we punish them, we lose our word-of-mouth growth."
They agreed to a compromise: implement Option A immediately as a stopgap, but start building Option B the same week. The throttle would stay at 100 requests per minute for two weeks, then move to the new burst-friendly system.
The two weeks that tested the plan#
The immediate throttle did what it was supposed to: infrastructure costs flattened. But the user response was worse than expected. In the first week, 312 free-tier users canceled their accounts. Another 47 posted on social media that QueryHub had "pulled a bait and switch." The support team, which consisted of two people, worked 12-hour days answering tickets.
On day 10, a developer named Marcus, who ran a small SaaS product on the free tier, published a blog post titled "QueryHub just broke my app." The post got 40,000 views. The CTO called Marcus directly and explained the situation. Marcus agreed to a temporary workaround: his account would get a manual exception until the new system was ready. That conversation convinced the CTO that the two-week timeline was too slow. She pulled two engineers off other projects to accelerate the build.
The decision point#
On day 14, the new rate limiter was ready for a test. It used a token bucket algorithm: each user got a bucket of 1,000 tokens that refilled at 100 tokens per minute. A short burst of up to 1,000 requests would be allowed, but sustained usage above 100 per minute would drain the bucket and then throttle. This was more forgiving for the bursty workloads that many developers had, while still protecting the shared infrastructure.
The founders met again. The question was whether to roll it out gradually or all at once. A gradual rollout would take another week and risk more cancellations. An all-at-once rollout could crash the system if the algorithm had a bug. They chose a middle path: roll out to 10 percent of users on day 15, monitor for 24 hours, then expand to everyone if error rates stayed below 1 percent.
The outcome#
The rollout went smoothly. Error rates stayed near zero. Over the next month, cancellations slowed to a trickle, and about 60 of the 312 users who left returned. More importantly, the number of paid conversions increased by 18 percent, not because of the throttle, but because the new system made the free tier feel fair again. Developers could see exactly how many tokens they had left, and the documentation explained the limits in plain language.
QueryHub did not become a breakout success overnight. But the episode taught the team a lesson they still repeat in onboarding: when you have to say no to users, say it clearly, say it early, and give them a path forward. The alternative, a silent throttle that breaks apps without explanation, costs more in trust than it saves in cloud bills.
More articles