The 14-Millisecond Delay That Exposed a Logging Bug in Production
Archi4 min read
The Bug That Hid in Plain Sight#
When a logging pipeline starts dropping messages, most teams look at volume, disk space, or network throughput. Priya Raman, a staff site reliability engineer at a mid-sized fintech company, looked at something smaller: a 14-millisecond gap between two timestamps. That gap led her to a bug that had been quietly corrupting log data for seven months. We spoke with her about the investigation, the fix, and why the smallest detail mattered more than any dashboard.
Q: How did you first notice something was wrong?
A: A support engineer asked why a customer's failed payment showed up twice in our internal dashboard. The dashboard pulls from logs. I expected a duplicate event, maybe a retry. When I pulled the raw log lines, the two entries had different request IDs but identical payloads. That should not happen. The only difference was a 14-millisecond gap between the ingestion timestamp and the event timestamp. Fourteen milliseconds is nothing to a human. It was everything to the pipeline.
Q: So the gap itself was the clue?
A: Exactly. Our log shipper stamps a time when it receives a message and the application stamps a time when it writes one. Under normal load, the gap is under 3 milliseconds. At 14 milliseconds, the shipper's internal buffer was flushing early. It was a sign that the shipper was under memory pressure, not the application. Everyone had been watching application CPU. Nobody watched the shipper's buffer.
Q: What did the early flush cause?
A: When the buffer flushes early, it splits a batch. The second half gets a new batch ID. Our downstream deduplication logic keys on batch ID plus event timestamp. Split batches created two entries with the same payload and slightly different timestamps. Over seven months, roughly 0.8 percent of our log events were duplicated. That sounds small until you multiply it by the volume. It skewed our error-rate alerts and inflated one customer's reported failure count by about 12 percent.
Q: How did you confirm it was the buffer and not something else?
A: I ran a controlled test. I replayed a fixed set of 10,000 log lines through the shipper with the buffer size at default and then at double. The default run produced 81 duplicates. The doubled run produced zero. That was the whole proof. No code change, no config overhaul. One buffer parameter. The fix was a one-line change to the shipper config, plus a note in the runbook.
Q: Why did it take seven months to find?
A: Because the symptom was a duplicate dashboard entry, and the dashboard was considered a secondary tool. The primary alerting system read from a different path. The duplication never triggered a page. It only showed up when a human looked closely at a specific customer case. The lesson is not that we need more dashboards. It is that we need to check the gap between two timestamps that should be close. That gap is a health signal, and we were not treating it as one.
Q: What changed after the fix?
A: We added a single metric: the p99 difference between ingestion time and event time. If it crosses 8 milliseconds, we get a warning. It has fired twice in four months, both times before any customer noticed. The other change was cultural. In postmortems, we now ask for the smallest observable difference, not the biggest failure. The biggest failure was a duplicate entry. The smallest difference was 14 milliseconds. The small one was the cause.
Q: What should other teams take from this?
A: Look at the parameters you never tuned. Our buffer size was a default from a library we adopted three years ago. Nobody had a reason to change it, so nobody looked at it. Defaults are not neutral. They are decisions made by someone else for a different workload. The 14-millisecond gap was the default showing its seams. Once we saw it, the fix took an afternoon. Finding it took seven months because we were watching the wrong number.
Q: Is there a broader pattern here?
A: In my experience, the bugs that survive longest are the ones that produce plausible output. A crash gets fixed in hours. A duplicate log entry looks like a retry. A 14-millisecond delay looks like normal jitter. The system keeps working, just slightly wrong. The only defense is to measure the things that should be constant and treat a change in them as a signal, even when nothing is on fire.
Q: One last question. What is the smallest detail you check first now?
A: The gap between when a thing happened and when we recorded it. If that gap moves, something in the middle is struggling. It does not matter whether the middle is a buffer, a network hop, or a queue. The gap tells you where to look before the dashboard tells you something is broken.
More articles