A child does not care how many APIs are involved. They notice only that the teddy did not answer.
Our first Tell Me Ted prototype used a chain that will be familiar to many AI teams. Record a short clip. Transcribe it. Send the text and page context to a model. Generate a response. Convert that response to speech. Store the audio. Download it. Play it.
Nothing about that design was absurd. It matched the services available, produced clear boundaries and let us test each capability independently.
It also created the wrong rhythm for the product.
The interaction we wanted
The ideal reading session is easy to describe from the child’s perspective.
The child starts Ted, opens a book and begins a session. Ted sees the first page when asked, reads it and then waits. The child can speak, interrupt after Ted finishes, turn the page and continue. If the child says “next page”, Ted asks for another image. If the speech is missed, a physical button provides a predictable fallback. If the network fails, the session ends clearly rather than getting stuck in an ambiguous state.
The child should not need to know whether speech recognition, OCR, language generation or text-to-speech happened. The state machine should be invisible.
That experience became the test for the architecture.
Where the chained path lost time
Latency accumulated in places that looked harmless in isolation.
The device had to finish recording before it uploaded audio. A cloud function waited for transcription before another model call could begin. Generated text then waited for speech synthesis. The resulting asset had to be stored, a URL returned and the audio downloaded before playback.
Each boundary added serial work, network overhead and another place to retry. State was split across multiple requests. Even if every individual service was “fast enough”, the end-to-end turn could still feel slow.
Voice interaction is unusually sensitive to this. In a graphical interface, a spinner can reassure an adult that work is happening. A young child sitting beside a silent teddy simply experiences the pause.
The problem was not only speed. Chained requests encouraged command-shaped interaction. Every turn had a beginning and an end defined by an API call. Interruption, continuing context and page changes all required additional orchestration.
The realtime redesign
The current direction creates one protected reading session rather than restarting the interaction for every capability.
The device establishes a time-bounded session through the backend, while long-lived service credentials remain off the device. The session maintains short-lived conversational context, requests a page only when needed and can begin returning speech before every downstream step has completed.
That reduces hand-offs and gives the conversation one clear owner. It also shifts measurement toward the experience: how quickly useful audio begins, whether turns complete cleanly and whether the system recovers predictably.
Realtime does not mean zero latency. It means the architecture can begin useful work and playback earlier, while keeping the context of an active session.
Half-duplex by design
The current interaction deliberately separates speaking and listening.
After a response finishes, the device opens a bounded opportunity for the next child turn. If no one speaks, the system becomes idle rather than generating another response.
This is less magical than full-duplex conversation, but it is easier to make predictable on constrained hardware. It reduces the immediate echo-cancellation problem and makes the device state clearer.
It is also a reminder that the most advanced technical option is not automatically the right prototype choice. We need evidence from the real device before adding complexity to make the interaction look more human.
Page turns are product events
The most interesting part of the redesign may be the humble page turn.
In the old flow, requiring a wake word for every page was easy to understand technically. It created an explicit boundary around each request. In use, it made the child restart the interaction again and again.
In the session-based flow, a clear verbal page turn can become a request for fresh page context. The system waits for that context before continuing to read.
A physical control provides a predictable fallback and duplicate inputs are handled conservatively. The product behaviour matters more than exposing the internal message flow.
The camera remains off unless a page is actually needed. That matters for privacy, power and trust.
Buffering and perceived quality
Streaming introduces different trade-offs. If the device plays every tiny audio chunk immediately, network jitter can create gaps. If it accumulates a large buffer, playback becomes smoother but the child waits longer and the firmware consumes more memory.
The right balance cannot be chosen from a cloud simulator alone. It depends on the audio path, network conditions, scheduling and memory pressure of the actual device.
This is one of the points where software habits meet physics. A buffering change is simultaneously a latency decision, an audio-quality decision and a memory-budget decision.
Failure and recovery are part of the interaction
A reading product must be designed for the page that cannot be read, the Wi-Fi connection that drops and the child who says something the system does not understand.
The session therefore has bounded lifetime and explicit ways to stop. Page context is used only as needed for the active interaction, and interruption, low battery or inactivity returns the device towards a safe idle state.
The child-facing response still needs refinement. A technically correct network shutdown is not an understandable experience. The product must distinguish “I could not see the page”, “I am offline” and “I need charging” in a way a child and parent can predict.
What we are measuring
We care about more than average server response time. The useful measures are closer to the interaction:
- time to first useful audio;
- total turn-completion latency;
- successful child turns after playback;
- successful verbal and button page changes;
- duplicate or failed page captures;
- recovery after a network interruption; and
- subjective conversational quality on the actual prototype.
We are not yet publishing a dramatic “before and after” number because the hardware flow and page-turn work are still being tested. The architectural direction is implemented. The product-quality evidence is still being gathered.
That distinction is the real lesson. Realtime is not a badge. It is a means of making the interaction feel less like a pipeline and more like reading together.
Next in the series: why an always-connected technical design is not automatically an acceptable design for children.
Tell Me Ted is an independent, pre-launch project that I work on outside my role at HubSpot. The views expressed here are my own and do not represent HubSpot.