The 1-Byte Flaw That Broke a Signed Firmware Update
Archi4 min read
Contents
The update that stalled at 40 percent#
On a Tuesday morning in March, an IoT vendor pushed a firmware update to roughly 12,000 field devices. By 9:40 a.m., the rollout dashboard showed 40 percent complete. By 10:15, it was frozen. Devices that had installed the update reported a boot loop; devices still waiting reported signature verification failures. The vendor pulled the package within the hour, but the damage was done: 4,800 units bricked, and a support queue that took nine days to clear.
The postmortem blamed the signing server. That was wrong. The signing server had done exactly what it was told. The failure traced back to a single byte in the update manifest, a length field that said 255 when the parser expected 256. The parser trusted it, read one byte past the signature block, and handed a malformed buffer to the verifier. The verifier rejected the signature. The bootloader, which had no fallback path, treated rejection as corruption and locked the device.
Where the byte came from#
The update manifest used a compact binary format. Each field started with a one-byte length prefix, followed by that many bytes of data. The signature field was the last entry, and its length prefix was supposed to be 0xFF, a sentinel meaning "read to the end of the block." A recent change to the manifest builder had started emitting the actual signature length instead, which for most devices was 256 bytes. The builder wrote 0x00 into the length prefix because 256 does not fit in one byte, then wrote the signature. The parser read 0x00, concluded the field was empty, and moved on. The verifier got a truncated buffer and rejected it.
That is the whole bug. One byte, written as zero instead of 0xFF. No memory corruption, no remote code execution, no attacker. The vendor's security team spent the first two days looking for a compromise that never happened.
The smallest detail that mattered#
The interesting part is not the off-by-one. It is that the format had two valid ways to encode the same field, and neither the builder nor the parser documented which one was correct. The builder change passed code review because the reviewer assumed the parser handled explicit lengths. The parser passed its tests because the test fixtures used the older builder. The integration test ran on three devices, all of which happened to have signatures shorter than 256 bytes, so the one-byte prefix worked by accident.
A single test device with a 256-byte signature would have caught it. A single comment in the format spec would have prevented it. Neither existed.
Why the rollout made it worse#
The vendor staged the update in three waves: 10 percent, 40 percent, then the rest. That is a reasonable pattern, and it is why only 4,800 devices failed instead of 12,000. But the staging logic checked in with the signing server after each wave and re-fetched the manifest. The manifest had been regenerated between waves, and the second generation had the same bug. The staging system never compared the manifest hash across waves, so it happily pushed the same bad byte three times.
A hash comparison is cheap. It would have stopped the second wave. It would have flagged the third before any device saw it.
The recovery path that did not exist#
Every bricked device had a recovery mode, but recovery required a signed image, and the only signed image available was the bad one. The vendor had to ship physical USB programmers to four regional depots and walk field technicians through a manual flash. That took nine days and cost an estimated $340,000 in labor and lost service credits.
The recovery path had been tested exactly once, during initial manufacturing, with a known-good image. Nobody had tested it with a rejected signature. That is the second small detail: a recovery mechanism that only works when nothing is wrong is not a recovery mechanism.
What to check this week#
, Find every place your code parses a length-prefixed field. Ask whether the length is explicit, implied, or both, and whether the spec says which., Add a test fixture with the boundary value. If a field uses a one-byte length, test with a payload of exactly 255 and exactly 256 bytes., In your staged rollout, compare the artifact hash at every wave. If the hash changes, stop., Test your recovery path with a deliberately corrupted artifact. If recovery needs a valid signature, you have a single point of failure.
The vendor's public postmortem, published three weeks later, runs to 14 pages. The fix is four lines: a bounds check in the parser, a spec update, a hash comparison in the staging tool, and a recovery image signed with a separate key. The bug was one byte. The lesson is that one byte is enough, and the systems around it should assume so.
More articles