The AI Coding Assistant Setting Most Teams Get Wrong: Context Window Size
Archi3 min read
Contents
The setting most people never open#
When a team adopts an AI coding assistant, the first question is usually which model. The second is how much it costs per seat. Almost nobody opens the settings panel and looks at context window size, the token budget the tool uses to decide how much of your project it reads before suggesting a line of code.
That number has quietly become one of the most consequential knobs in a developer tool, and most ordinary users leave it at whatever the vendor picked. My view: for the typical developer on a typical project, the default is too large, and changing it today will save money, reduce strange suggestions, and make the tool easier to trust.
What the context window actually does#
Every assistant works from a fixed budget of tokens. Tokens are roughly three quarters of a word. A 128,000 token window can hold a short novel. When you ask for a function, the tool fills that budget with whatever it thinks is relevant: open files, recently edited files, project structure, maybe an index of the whole codebase.
The sales pitch is straightforward: more context means the model sees more of your code and makes fewer mistakes. In practice, the relationship is not linear. Models attend unevenly across long inputs. Details buried in the middle of a large prompt are the easiest to miss. A 200-line file that matters can get diluted by 2,000 lines that do not.
Why bigger defaults cost you twice#
First, the bill. Most hosted assistants charge by input tokens. If your tool ships 100,000 tokens per request and you make 200 requests a day, you are paying for 20 million tokens daily, much of it boilerplate your project already contains. Multiply across a team and the invoice is real. The same pattern shows up on cloud bills generally: defaults favor capability over economy, and the user pays the difference.
Second, the output. Large contexts invite lazy prompts. If the tool will figure it out from the whole repository, why describe the function you want? The result is a suggestion that is plausible, compiles, and slightly wrong, because it was built from a vague request plus a pile of loosely related code. Shrinking the window forces you to say what you mean.
What to change today#
Start with one project and one afternoon.
Check whether your tool reports token usage per request. Many do, in a status bar or a log. Look at the number for a request where you already know the answer. It is often five to ten times larger than the code you actually needed.
Find the setting that controls automatic file inclusion. Names vary: context size, retrieval depth, workspace indexing, related files. Turn off automatic inclusion of the entire project and keep it to open files or an explicit selection.
Write prompts that name the files. "Using only src/billing/invoice.ts, change the rounding to banker's rounding" is a better request than "fix rounding." It gives the model a small, precise context and a clear boundary.
If your tool supports a project rules file or a short instruction document, keep it under a page. Long instruction files consume the same budget as code and are often ignored in the middle.
The exception: when large context helps#
There are real cases for a wide window. Migrating a pattern across 40 files benefits from seeing all of them. Answering a question like "where is this environment variable read?" needs broad retrieval, not narrow focus. The mistake is treating that as the default for every request. A useful habit is to keep the window small for edits and switch to a larger setting for exploration, then switch back.
For teams with a shared configuration, set the default low and document the cases where people should raise it. That is a one-line change in most tools and it changes behavior more than swapping models.
A note on what gets sent#
Smaller context also reduces what leaves your machine. A tool that reads every file on save is exporting far more of your codebase than one that reads the file you asked about. If your team has any policy about source code leaving the network, the context setting is the first place to look, before the model choice.
None of this requires a new vendor or a procurement cycle. It requires opening a settings page that most users have never seen, reading one number, and deciding it should be smaller. The assistant will get less to work with, and in most cases it will do better work because of it.
More articles