The AI Coding Assistant Decision: A 1,200-Person Firm's Deployment Checklist

Archi3 min read

aipolicychecklistsoftware

Contents
  1. The decision on the table
  2. First, define the data boundary
  3. Second, confirm the training setting
  4. Third, limit the blast radius
  5. Fourth, measure quality without losing context
  6. Fifth, write the incident path
  7. Sixth, prepare the exit
  8. Seventh, train the people
  9. The decision

The decision on the table#

A 1,200-person software company in the logistics sector had to decide whether to let a vendor coding assistant read two of its repositories. One repository held customer-facing code. The other held internal billing logic. The assistant would suggest code in the IDE and in pull requests. The vendor's contract said data could be used to improve the service unless the customer opted out. The company's AI policy, approved three months earlier, said model training on company data required written approval. No one had tested the assistant.

The decision was not binary. The company could approve a pilot, approve with limits, or reject the vendor. It chose to run a two-week checklist. The checklist had 12 items across legal, security, engineering, and operations. Each item had an owner and an evidence file.

First, define the data boundary#

The legal team asked for a list of code categories. The list had to name what could be sent, what could be stored, and what could be deleted. The company drew three lines.

Public open source code: allowed.

Internal code with no secrets: allowed after review.

Customer data, encryption keys, and deployment credentials: blocked.

The vendor's browser extension could not read .env files. The company added a policy check in the CI pipeline that failed if a secret pattern appeared in a diff. This was not a complete fix. It was a tripwire.

Second, confirm the training setting#

The contract had a default. The default allowed the vendor to use prompts and code snippets to train models. The company turned that off. It asked for written confirmation. It also asked for the retention period. The answer was 30 days for prompt logs. The security team wanted 0 days for code content. They settled on 7 days, with deletion before the next training cycle.

Third, limit the blast radius#

The pilot covered 12 engineers. They worked on two services. One service handled route calculations. The other handled invoice generation. The company gave each engineer a test account. The assistant could not access production logs. It could not open tickets. It could not change infrastructure.

The platform team added a kill switch. If the vendor missed a security patch deadline, the plugin would be disabled by policy. The switch was tested in a staging environment.

Fourth, measure quality without losing context#

The engineering leads defined five checks for the pilot.

  1. Build success rate for generated code.
  2. Number of security findings in generated diffs.
  3. Review comments per pull request.
  4. Time from task assignment to first review.
  5. Engineer confidence, collected with a short weekly form.

The company did not use a single score. It wanted to see where the assistant helped and where it added work.

Fifth, write the incident path#

The security team asked what would happen if the assistant produced a vulnerable dependency. The answer had to be fast. The company added the assistant to its incident register. A generated file with a known CVE would trigger a patch ticket within one business day. The vendor had to share incident details within 72 hours. The contract was amended.

Sixth, prepare the exit#

The legal team asked for a data export and deletion plan. The company required a full list of stored artifacts, a deletion timeline, and a contact for data subject requests. The vendor agreed to delete prompt logs on request. The company kept a copy of the final report.

Seventh, train the people#

The company ran a 45-minute session for engineers. It covered three rules. , Do not paste customer identifiers into prompts., Treat every generated block as untrusted code., Mark generated code in the pull request description.

The session was short. The rules were specific.

The decision#

After two weeks, the company approved the assistant for the two services. It limited the pilot to 12 engineers. It kept the training opt-out. It set a 30-day review. The checklist was stored in the company's AI policy repository. The vendor's contract was updated.

The company did not make a permanent decision. It made a reversible one. That was the point. The checklist did not answer every question. It made the next question easier.

More articles