The refactor worked. The code was dramatically simpler. One problem: a lifecycle-critical maintenance capability no longer survived the change intact.
For a web service, a missed deployment detail is painful but often reversible. For a physical product, recovery and maintainability are part of the product itself. If either disappears, the next fix can become much harder to deliver safely.
We caught the issue during development. Follow-on changes restored the capability and added automated checks so the same class of regression is harder to repeat.
The incident still changed how I think about coding agents.
Like many engineers, I began using AI coding tools for small jobs: explain a function, draft a test, fix a type error, suggest a refactor. The results were fast and easy to review.
As the tools improved, the tasks became broader. Codex could inspect the repository, edit multiple components, run tests and return a coherent change. That made it natural to delegate work that looked less like autocomplete and more like an engineering ticket.
The always-Realtime refactor was exactly the kind of job where that capability is compelling. The existing firmware state machine contained months of accumulated flows. The product direction had become clearer: the MVP should enter one persistent reading session rather than preserve several legacy experiences as equal first-class paths.
The resulting simplification removed a great deal of old state and made the main interaction easier to reason about. Locally, the change made sense.
The affected capability was not the feature being refactored. It crossed device startup, connectivity, configuration and release operations. To a task focused on the reading experience, some of that machinery looked peripheral.
Our prompt described what should become simpler. It did not describe, with equal force, what must remain unchanged.
The tests had the same blind spot. They represented the new reading path better than they represented the complete lifecycle of a deployed device. A test suite can only protect the behaviours it encodes.
The agent therefore produced a change that was coherent inside the boundary we had drawn. The boundary was wrong.
For repository-level work, I now separate requirements into two lists.
The first describes the behaviour to change: what the user does, what the system should do and how success will be observed.
The second describes invariants: the behaviours and safety properties that may not change even if they are not visible in the immediate feature.
For Tell Me Ted, those invariants cover:
Writing these down does more than inform the agent. It forces the human owner to decide what the system actually promises.
Here is the structure I now use for material changes:
Explain which component owns the behaviour, the relevant entry points and the interfaces that already exist. Link to the current architecture and runbooks rather than restating them from memory.
Describe what a child, parent, operator or developer will observe. Avoid starting with the implementation you expect.
List lifecycle, security, privacy and resource constraints. State which adjacent features are explicitly out of scope.
Name the files or components that may change, and identify high-risk areas that require an explanation before modification.
Provide the exact build, unit, integration and acceptance checks. For embedded work, include a physical-device test and a resource comparison.
Require a summary of assumptions, deleted paths, untested behaviour, residual risks and rollback instructions.
That format is less clever than a long prompt full of model-specific techniques. It works because it expresses engineering ownership.
After repairing the lifecycle issue, we encoded the rule in more than one place.
The repository now contains invariant checks that verify critical maintenance, sequencing and recovery behaviours remain present. The agent guide repeats the rule so future tasks encounter it before editing, and release work validates recovery behaviour.
This is an important progression:
Prompts are useful, but prompts alone are not controls. Durable knowledge belongs in code, CI, tests and architecture boundaries.
There is a useful organisational effect too. Once an invariant is expressed as a test, it stops depending on the one engineer who remembers why an apparently unrelated path exists. New contributors and future agents receive the same boundary. The repository becomes a better specification than the prompt that happened to start the change.
Agents make large diffs inexpensive. That means reviewers should pay unusual attention to what disappeared.
I now ask for a deletion-focused review: which functions, permissions, configuration values and operational paths were removed or bypassed? Was each deletion required by the task? What previously called it? Which tests stopped exercising it?
For embedded changes I inspect resource headroom and dependency changes. For cloud changes I inspect authentication, exposure, secrets and recovery. Then the complete flow is tested on representative hardware.
An agent reviewing its own diff is useful, but it is not independent verification. The human review and the tests remain the decision boundary.
It would be easy to turn this into a warning that coding agents are unsafe. That is not my conclusion.
Codex helped us execute a large architectural change faster than we could have done manually. The simplified system became a better foundation for the reading experience. The failure was that we delegated a task without fully specifying the system around it.
A coding agent can own the mechanics of a task. It cannot own the deployed devices, the families using them, the company’s security posture or the recovery plan. Those consequences remain with us.
The senior engineering skill is shifting. Producing code is becoming cheaper. Defining boundaries, protecting invariants and deciding what counts as evidence are becoming more valuable.
That is not less engineering. It is the part of engineering we should have been protecting all along.
Next in the series: a guided tour of the hardware that turns an AI service into a physical bear.
Tell Me Ted is an independent, pre-launch project that I work on outside my role at HubSpot. The views expressed here are my own and do not represent HubSpot.