Nothing in the first architecture was obviously wrong. That was what made it difficult to replace.
Tell Me Ted began with a set of clear capabilities. Transcribe speech. Generate a story. Chat. Extract text from a book page. Convert text to speech. Record a parent’s story.
The natural software design gave each capability an endpoint. The device captured an input, called the relevant cloud function, waited for a result and played an audio asset. Shared models kept request and response shapes consistent. Each service could be mocked and tested.
It was a sensible architecture for proving the parts.
It was not the best architecture for the interaction we eventually wanted.
The system mirrored the APIs
Early architectures often reflect the building blocks available at the time. When speech recognition, vision, generation and speech synthesis are separate services, drawing separate boxes for them feels like good separation of concerns.
The trouble appears when the product needs one continuous experience.
A child does not begin a “transcription flow” and then enter a “generation flow”. They say something while looking at a page and expect the same companion to understand the next moment. The conversational state crosses every service boundary we had created.
The architecture was organised around technical capabilities. The product was organised around a reading session.
How complexity accumulated
Each endpoint needed input validation, authentication, error handling, model calls, storage and logging. The firmware needed states for recording, processing, downloading and playback for several different modes.
Story creation and story reading shared some behaviour but differed in inputs. Chat needed turns and session context. Parent recording introduced a longer capture window and persistence. Each path acquired retries and edge cases.
The number of boxes was not the main issue. Ownership was.
Conversational state existed in several places and had to be reconstructed across calls. Error recovery could return the device to idle while backend work was still completing. Interrupting playback meant negotiating a path designed around completed assets.
Complexity multiplied at the seams.
The product symptoms
The first symptom was latency. Each sequential call added time before audio began.
The second was command-shaped interaction. A user action mapped neatly to one endpoint, so continuing the experience often meant reactivating it.
The third was fragile context. A page change or conversational follow-up had to be reconstructed across requests.
The fourth was difficult interruption. Once the system was waiting for a complete generated audio file, there was no natural streaming boundary.
These were not isolated performance bugs. They were evidence that the architecture’s unit of work did not match the product’s unit of experience.
The simplification hypothesis
We chose a more continuous reading path with one clear session owner.
The device establishes a protected, time-bounded session through the backend. That session holds only the context needed for the active interaction, can return speech progressively and requests a page only when needed.
The device software became centred on startup, connectivity, session setup, active reading and shutdown rather than preserving a separate first-class state graph for every earlier experiment.
This reduced hand-offs and clarified ownership. The session owns the conversation, the device owns hardware and lifecycle, and the camera remains event-driven.
Simplification moved complexity; it did not erase it
A continuous session still requires authentication, expiry, reconnect and cleanup. Progressive playback requires buffering. Short-lived context needs bounds. Child turns need detection and timeouts. Page changes require coordination across several components.
The embedded constraints did not disappear. Neither did network failure, child safety, privacy, maintenance or recovery.
That is the honest test for a refactor: not whether it reduced the line count, but whether the remaining complexity sits in components that can own it coherently.
The realtime design places more responsibility in the active session, which is exactly where conversational responsibility belongs.
A simple responsibility test
One way to evaluate the new design is to ask a failure question at each boundary.
If the camera cannot see the page, the system needs a bounded retry or a clear error. If reading context is lost, the session should request fresh context rather than silently use the wrong page. If the connection drops, short-lived state should be cleaned up and the device should return towards idle. If a maintenance service is unavailable, the core experience should not be blocked unnecessarily.
Those answers are not perfect yet, but each has an owner. In the original architecture, the same failure could cross several endpoints, storage operations and firmware states before anyone owned the complete outcome.
That is the kind of simplification worth pursuing: fewer places where responsibility can disappear between two boxes.
What we would reuse elsewhere
First, design from the interaction loop. Write down how the user starts, continues, interrupts, recovers and stops before drawing service boundaries.
Second, measure end to end. A dashboard full of individually healthy services can still hide a poor user experience.
Third, give state one clear owner. Duplicated or implicit state creates more failure modes than explicit communication.
Fourth, keep lifecycle concerns outside feature refactors but inside system invariants. Boot, update, rollback, auth and safe shutdown must survive architectural simplification.
Fifth, preserve explicit fallbacks. Realtime inference does not make a physical page button obsolete. It makes the button a reliable event in a better session model.
Finally, revisit architecture when platform capabilities change. The first design was not foolish; it fit a moment in the available APIs and our understanding of the product. Refusing to reconsider it would have been the mistake.
Questions for an architecture review
If you are evaluating an AI system, ask:
- Does the architecture follow the user’s interaction or the provider’s API catalogue?
- Where does conversational state live?
- How many sequential boundaries sit before the first useful output?
- Can the user interrupt and recover?
- Which component owns cleanup after partial failure?
- Which old capabilities remain because they matter, and which remain because removing them is uncomfortable?
- What lifecycle and security invariants could be lost during simplification?
Our first architecture proved the capabilities. Rebuilding it taught us to make the reading session, not the API call, the centre of the system.
Next in the series: what software engineers learn when schematics, suppliers and prototype tolerances become part of the product.
Tell Me Ted is an independent, pre-launch project that I work on outside my role at HubSpot. The views expressed here are my own and do not represent HubSpot.