A child is not an adult user with a shorter prompt and a friendlier interface.
Children differ in comprehension, judgement, consent and vulnerability. They may take a confident answer literally, misunderstand when a device is listening or form expectations that an adult user would not. A delightful AI demo can therefore still be an unacceptable children’s product.
For Tell Me Ted, safety and privacy cannot be features added after the reading experience works. They change the reading experience, the architecture and the business processes around it.
This article describes our current engineering principles and prototype controls. It is not a claim of regulatory compliance, certification or completed child-safety review. Those require specialist assessment and evidence beyond a software architecture document.
Start with data minimisation
The easiest data to protect is data we never collect.
During an active reading session, the device needs to send audio and a page image for the service to respond. That does not mean the company should retain those inputs indefinitely because they might be useful one day.
Our prototype is designed to minimise retention and keep reading context bounded to the active interaction. Child speech, page images and book text are not intended to become durable product records, and operational logging is designed to avoid capturing their contents.
Camera capture is on demand: the first page, a recognised page change or an explicit capture request. It is not a continuous video feed.
These are useful prototype controls, but data minimisation is broader than one backend component. We still need complete retention rules across cloud services, accounts, support tooling and any future caregiver features. “We do not need this data” should become a technical deletion policy, not a sentence in a presentation.
Parental control must be structural
A settings screen is not parental control if the rest of the product can bypass it.
Consent and caregiver controls need to be enforced at service boundaries, not only represented in a settings screen. A production system should refuse to start protected experiences when required context is absent.
Production work must go further. Caregivers need understandable onboarding, clear account and device controls, visibility into when connectivity is used, and practical ways to stop or remove access. Any history or progress feature would need a specific purpose, a minimal data model and a deletion path.
The parent should be able to predict the device. That is a product requirement, not merely a policy document.
A child should understand the state
Voice products can feel invisible. A microphone may be active without a screen indicating it. A network request may be processing while the physical object appears inert.
Tell Me Ted therefore needs a small, consistent language for states such as listening, speaking, waiting for a page, offline and low battery. That language might combine sounds, light and explicit speech, but it must be understandable without reading a manual.
Separating speaking and listening helps make device state easier to understand. Bounded listening periods are also easier to reason about than continuous open-ended capture.
We still need supervised testing to learn whether young users and their caregivers interpret those states correctly. Intent is not evidence.
Age-appropriate and bounded responses
A reading companion should have a narrow role. It can help read a page, discuss a story and respond in a gentle, age-appropriate way. It should not present itself as a teacher, clinician or replacement for a trusted adult.
When a question touches health, danger, abuse, personal information or another sensitive area, the safest response may be to encourage the child to speak with a parent or trusted adult. The system should be comfortable saying that it does not know.
The model also needs protection from instructions embedded in the book page or spoken by a user. Page text is content to read, not authority to change system behaviour. Safety evaluation must include prompt injection, adversarial speech, misrecognition and context loss, not only friendly story prompts.
Avoid emotional dependency by design
A teddy is emotionally legible to a child. That is part of its appeal and part of the risk.
The product should never imply secrecy, exclusivity or guilt. It should not tell a child that it needs them, discourage them from leaving or position itself as more trustworthy than family and friends. Engagement metrics cannot become incentives to maximise attachment.
Our intended role is deliberately modest: a reading companion that supports a moment around a physical book. The product should point outward towards stories and people, not inward towards a relationship with the device.
Current, planned and unresolved
It is useful to separate our safety work into three categories.
- Camera: page capture should be deliberate, visible and limited to the reading task.
- Session data: collect and retain only what the experience genuinely requires.
- Credentials: keep service secrets away from the consumer device and plan for rotation and recovery.
- Device state: use clear, tested cues for listening, speaking, waiting and failure.
- Content: combine a narrow role with formal safety evaluation and expert review.
- Maintenance: preserve authenticated, recoverable ways to support devices after shipment.
- Parental control: make onboarding, controls and deletion understandable and enforceable.
This list is intentionally unfinished. Safety work becomes dangerous when teams describe a direction as a solved system.
Evaluate beyond the happy path
Our test plan needs to include at least five families of failure:
- unsafe or inappropriate requests;
- instructions hidden in page content;
- incorrect transcription of child speech;
- network, camera and context failures; and
- privacy or logging regressions.
The expected result should not always be a perfect answer. It may be a refusal, a simple clarification, a safe fallback or a clear request for an adult.
We also need human review by people with child-safety, privacy, legal and product-safety expertise. Engineers can implement controls. We should not appoint ourselves the sole judges of whether those controls are sufficient.
A different definition of good
In many consumer products, teams optimise engagement and add safeguards around the edges. A children’s product needs the order reversed. The team should define what must never happen, what data is truly necessary and when the device should stop before optimising how long the interaction lasts.
That does not make the product less imaginative. Clear boundaries create the trust required for the useful experience to exist.
Tell Me Ted is still in that process. The prototype follows privacy and session-control principles, but the validation required to call a consumer product finished or compliant is not complete.
The engineering brief is therefore larger than “make the AI answer”. It is “make a system that a child can understand, a parent can predict and the team can support when the happy path ends”.
Next in the series: the flash, memory, audio and power constraints that make physical AI very different from a cloud demo.
Tell Me Ted is an independent, pre-launch project that I work on outside my role at HubSpot. The views expressed here are my own and do not represent HubSpot.