The Daily Build: A Developer's Toolkit Then and Now
Archi3 min read
Contents
Five Years in the Life of a Build#
In 2019, a typical morning for a backend developer meant waiting. Waiting for dependencies to download, for tests to run, for a colleague to review a pull request. Today, the same developer might ship a dozen small changes before lunch. The tools changed, and so did the rhythm of the work. We asked a staff engineer at a mid-sized payments company to walk us through what actually shifted.
The Feedback Loop#
Q: What is the biggest difference you notice day to day?
A: Speed of feedback. In 2019, a full CI run on our monolith took 45 minutes. If you broke a test, you found out after a long coffee. Now the same suite finishes in about seven minutes, and most of that is spent on integration tests that we cannot parallelize. The editor itself catches type errors before I save the file. That changes how I write code. I experiment more because the cost of a mistake is lower.
Q: Does that make you a better engineer?
A: It makes me a faster one. Whether it makes me better is an open question. When the feedback loop was slow, I spent more time reasoning about the code before I ran it. Now I often let the tools tell me what is wrong. I have to remind myself to think first.
The Shape of the Stack#
Q: How has the stack itself changed?
A: Five years ago, we ran maybe 15 services. Today it is closer to 60. Each one has its own repository, its own pipeline, its own set of alerts. The tooling for managing that sprawl has improved, but the sprawl is real. We spend more time on the seams between services than on the services themselves. In 2019, a bug was usually in one place. Now it is often in the contract between two places.
Q: What about the local development environment?
A: That is where the biggest convenience win landed. In 2019, getting a new hire productive took three days of installing databases, message brokers, and language runtimes. Now we hand them a container configuration and they are running tests within an hour. The tradeoff is that fewer people understand what is inside the container. When it breaks, the debugging starts from a higher level of abstraction.
Testing and Confidence#
Q: Are you more confident in your releases than you were five years ago?
A: More confident, yes, but for different reasons. We deploy smaller changes more often, so the blast radius of any one release is smaller. The tools also give us better observability: traces that used to require a dedicated team to set up are now a library call away. What worries me is that the test suite itself has become more complex. Flaky tests are still a tax. We probably spend as much time managing test reliability as we did in 2019, just on a larger surface.
The Cost of Convenience#
Q: What has gotten worse?
A: Configuration. The number of knobs has exploded. Five years ago, a service had a config file with maybe 20 settings. Now it has 200, spread across environment variables, feature flags, and a remote config service. Nobody understands all of them. We have had outages caused by a single flag set to the wrong value in one region. The tools made it easy to add knobs and hard to know which ones matter.
Q: Any advice for teams feeling that pain?
A: Treat configuration like code. Review it, test it, and delete what you do not use. We started a quarterly cleanup of feature flags last year. It felt like busywork at first. It cut our config-related incidents by half. That is a small number, but it is real.
What Has Not Changed#
Q: What is exactly the same as it was five years ago?
A: People still want clear error messages. A stack trace that points to the wrong file is just as frustrating now as it was then. Documentation still goes stale. And the hardest part of the job is still deciding what to build, not how to build it. The tools have removed a lot of friction from the how. The what is still a human problem.
More articles