Field Notes: The AI Logging Decision That Cut a Company's Audit Risk

Archi4 min read

aipolicyauditlogging

Contents
  1. The Choice in Front of Us
  2. What We Tested
  3. The Audit Query That Decided It
  4. Where the Hashed Path Broke
  5. The Cost of Getting It Wrong
  6. What We Would Do Differently
  7. The Numbers After Two Weeks

The Choice in Front of Us#

On a Tuesday morning in March, the engineering lead at a 180-person fintech put two architecture diagrams on the wall. The company was preparing for an internal AI governance review and had to decide how much of its customer support assistant's activity to log. Option A: store full prompts and completions for 90 days. Option B: store hashes, token counts, and decision metadata only.

The legal team wanted full records for audit defensibility. The security team wanted less data at rest. The finance team wanted the storage bill below $400 a month. We spent two weeks testing both paths against real traffic shapes. This is what we found.

What We Tested#

We built a shadow pipeline that mirrored 8% of live support traffic, about 3,100 conversations a day. Each path ran for five business days. We measured storage growth, retrieval speed for a simulated audit query, and the number of fields a compliance reviewer could reconstruct without asking an engineer for help.

The full-log path averaged 1.9 KB per conversation after compression. That is small until you multiply by 30 days and 38,000 daily conversations across the whole system. The hashed path averaged 240 bytes, mostly because token counts and model version strings dominate the record.

The Audit Query That Decided It#

We wrote a realistic reviewer request: show every conversation in February where the assistant gave a refund amount above $200 and the customer had previously disputed a charge. On the full-log path, the query returned 214 rows in 1.4 seconds. On the hashed path, it returned 0 rows because the refund amount lived inside the completion text, not in metadata.

That result was not a surprise, but it forced a third design. We added structured extraction: the application now parses refund amounts, escalation flags, and policy citations into typed columns before the log is written. The raw prompt and completion get dropped. The extracted fields stay.

This is the decision the company actually made: keep the facts a reviewer needs, not the conversation that produced them. Storage dropped to roughly 310 bytes per conversation. The February query returned 214 rows in 0.8 seconds because the database could use an index instead of scanning text.

Where the Hashed Path Broke#

The hashed-only design failed a different test. A reviewer asked to confirm that a specific customer's transcript had not been altered after the fact. A hash proves integrity only if you still have the original to compare. Without the prompt stored anywhere, the hash is a receipt for a document nobody kept.

The fix was narrow. The company now stores a salted hash of the prompt alongside the extracted fields, and the salt rotates quarterly. If a transcript surfaces from another system, such as a customer screenshot or a support ticket export, the hash can confirm it matches what the model saw. The model output itself is not retained beyond 24 hours in a quarantined bucket with access limited to two named engineers.

The Cost of Getting It Wrong#

The first draft of the policy said logs would be kept for 90 days. Legal read that as 90 days from the end of the customer relationship. Engineering read it as 90 days from the request. The gap matters because a dispute can arrive eleven weeks after a conversation. We added a retention field to the schema and a scheduled job that recalculates the expiry whenever a ticket reopens. That single field change took four hours and prevented an untested assumption from reaching the review.

What We Would Do Differently#

Start with the reviewer's question, not the storage format. We spent three days comparing byte counts before anyone wrote down what an auditor would actually ask. The extracted-field design came from that question, not from a cost model.

Test the deletion path before the write path. The company's first purge job silently skipped rows where a nullable column was empty. We caught it on day four because a test fixture inserted a conversation with no customer ID. A reviewer would not have found that until an audit.

Put the model version in every row. During testing, a provider pushed a minor update that changed refund phrasing. Without a version string, the February query could not distinguish outputs from two model builds. The version column is now part of the primary key for audit views.

The Numbers After Two Weeks#

Storage cost settled at $180 a month for the full traffic volume, below the $400 target. Retrieval for the standard audit query stayed under one second at the 95th percentile. The company kept 14 structured fields per conversation and dropped the raw text. The decision was not between privacy and auditability. It was between storing a conversation and storing its consequences.

More articles