One Missing Semicolon in a Log Parser: A Data Breach Q&A on the Smallest Detail
Archi4 min read
Contents
- The Detail That Decided It
- The attack did not start with a clever exploit. What was the first sign something was wrong?
- Why not? You had logging.
- How did the attacker find that?
- What did the dropped lines contain?
- What finally surfaced it?
- You are arguing that the smallest detail is often a parser rule. Is that a fair summary?
- What would you change first if you went back?
- What about the default password on staging?
- How should teams test for this kind of blindness?
- What is the lesson for someone who runs a smaller operation?
- One last thing. What did the postmortem title say?
The Detail That Decided It#
In a breach that exposed 1.8 million customer records, the entry point was mundane: a web application firewall rule that stopped matching after a vendor update. But the detail that let attackers roam undetected for 94 days was smaller still. A log parser silently dropped any line containing an unescaped pipe character. Attackers learned to include one.
I spoke with Dana Whitfield, a detection engineer at a mid-sized payments processor who worked the incident response. She asked that the company not be named. The conversation has been edited for length and clarity.
The attack did not start with a clever exploit. What was the first sign something was wrong?#
It started with a forgotten staging server that still had a default admin password. An attacker found it in 11 minutes. That server had a network path to a production database because a firewall rule from 2021 was never removed. So the entry was boring. The interesting part is that we did not see it for three months.
Why not? You had logging.#
We had lots of logging. We shipped 40 gigabytes a day into a SIEM. But the parser that turned raw web logs into fields had a rule: split on the pipe character. If a request line contained an unescaped pipe, the whole line failed to parse, and the pipeline dropped it. No error alert, no dead letter queue. The only clue was a small drop in volume that nobody watched.
How did the attacker find that?#
They probably did not know. Their tooling encoded the payload with a pipe by accident, then kept using it because it worked. That is the part that keeps me up. We spent weeks looking for sophisticated evasion. The answer was a parsing quirk that happened to be in their way and happened to help them.
What did the dropped lines contain?#
The requests that touched the staging server and then moved laterally. Without those lines, our timeline had a gap. We could see database queries, but not who made them. It looked like an internal job. For 94 days, our dashboards were green because the evidence never arrived.
What finally surfaced it?#
A customer called about a password reset email they did not request. That triggered a manual review. One analyst pulled raw logs from the load balancer instead of the SIEM and saw the pipes immediately. The fix was a two-character regex change. Finding the breach took a week after that.
You are arguing that the smallest detail is often a parser rule. Is that a fair summary?#
It is narrower than that. The smallest detail was a schema assumption: that a request line would never contain a pipe. That assumption was true for years. It became false when a vendor changed an encoding default. Nobody owned the assumption, so nobody rechecked it.
What would you change first if you went back?#
I would add a drop counter per parser stage and alert when it moves. That is not a new idea. But we had alerts on failed logins and database errors and nothing on dropped evidence. We also should have kept raw logs for 30 days, not seven. We were paying for storage we never used.
What about the default password on staging?#
That is the obvious fix and we did it. But notice the sequence. The default password was the entry. The firewall rule was the path. The parser was the blindfold. If you fix only the first, the other two remain. A breach is rarely one failure. It is a chain where each link looks minor.
How should teams test for this kind of blindness?#
Inject known bad lines into the pipeline and confirm they arrive. We now run a weekly canary: a synthetic request with pipes, quotes, and non-ASCII characters. If it does not show up in the SIEM within five minutes, we get paged. It caught two broken parsers in the first month after we deployed it.
What is the lesson for someone who runs a smaller operation?#
You do not need a 40-gigabyte pipeline to have this problem. Any script that splits on a delimiter has the same assumption. Check what happens when a field contains the delimiter. If the answer is that the line disappears, you have a blind spot. The smallest detail is usually the one nobody wrote down as a rule.
One last thing. What did the postmortem title say?#
It said "Staging Exposure and Data Loss." I wanted it to say "Pipe Character." The title matters less than the finding. But I still think the smaller title would have taught more people.
More articles