The first time we sat a toddler in front of Sisu and hit Hear, the picture was fine. The voice was not.

Sometimes "Apple" came out crisp. Sometimes the same tap said "capital A." Nepali words jumped between usable and nonsense depending on the phone. On a demo day you shrug. When the whole study screen is built around listening, that is the product failing.

Sisu is Codse's bilingual first-words app: English and Nepali, tap the art, hear the word. For a long time that path used on-device text-to-speech (expo-speech). This post is how we moved to bundled MP3s, what we built to keep quality from rotting, and the bugs that only showed up after the files were already in the app.

The tempting shortcuts

Two ideas show up immediately when on-device speech starts embarrassing you.

Call a cloud TTS API every time the kid taps Hear. Modern voices (we use ElevenLabs) sound great until a two-year-old hammers the button on a flight, a parent hits airplane mode, or your bill becomes a function of toddler enthusiasm. Hear has to feel instant. It cannot wait on a network round trip.

Record every word by hand. Fine for a dozen words. Not fine for English plus Nepali packs, male and female voices, letter names for spelling mode, and the next content drop.

We needed a third path: generate the audio once, review it like humans, ship the files inside the app.

Decide what "done" means before you generate

"Bake" here just means offline generation: run a script, get MP3s on disk, commit them. Before we wrote those scripts we locked a few rules.

Same voice on every phone (Sarah and James for both languages). Files ship inside the binary; on-device TTS stays only as a fallback if a clip is missing. A human listens before we regenerate the TypeScript asset map. And the app looks up audio by a stable key, lang|gender|text (for example en|female|Apple), not by whatever prompt string we used that Tuesday.

That last rule sounds boring. It is the difference between "tweak Mom's delivery" and "rename half the asset tree."

In practice the pipeline is:

1. Build a speak plan from lesson JSON (what to say, with light mood where it helps)
2. Generate hashed MP3s into assets/audio/… and write a manifest
3. Review / replace bad takes in Lesson Studio
4. Regenerate the require() map Metro uses to bundle the files
5. Play them on device through expo-audio

First bake: coverage is not quality

Three commands share one bake library:

We generate through ElevenLabs v3. English usually goes via Replicate's hosted elevenlabs/v3 model. Nepali prefers the ElevenLabs API directly when the key is set, otherwise Replicate. The codegen step writes src/generated/audio-assets.ts so React Native's Metro bundler can include every clip without scanning the filesystem at runtime.

The first full bake got us to roughly 1,730 clips (about 900 English and 830 Nepali across male and female). File coverage looked green. Listening did not.

Saying "Apple" once is easy. Spelling A P P L E is a different job. Bare letter prompts produced "capital P." Bare "Z" went mushy or British. Family words needed a little warmth; most vocab needed none.

We split three layers in a speak-manifest:

1. Catalog text — what the app looks up (Apple, आमा, p)
2. Speak text — what the model actually hears
3. Context fields (previous_text / next_text) — sent to the API for prosody only, never spoken aloud

Letter neighbors plus a "small letter" hint stopped the capitals. Z is forced to American "zee." Mood tags (warm, happily, excited, curious, softly) stay rare: family words, treats, loud animals. Toddlers need a clear spoken word more than theatrical delivery.

Then we hit the bug you only find with headphones on.

"Warm Mom," spoken out loud

On ElevenLabs' own eleven_v3 API, a prompt like [warm] Mom treats [warm] as direction. On Replicate's ElevenLabs v3 wrapper, those brackets were voiced. The clip literally said "warm Mom."

Docs will not save you here. Ears will.

So the prompt path became provider-aware. Direct ElevenLabs keeps the expression tags. Replicate strips them and maps mood to voice settings instead (stability, style, similarity_boost). Studio overrides follow the same rule. The catalog key never moves when you change how something is spoken.

Somewhere in the rebakes we also learned hashed folders fill with ghosts. Filenames are content hashes, so an old take does not overwrite a new one; it just sits there. One orphan prune deleted about 25MB of MP3s that were no longer in the manifest. If you rebake without pruning, you ship junk and wonder why the binary got fat.

Why Lesson Studio exists

A CLI bake answers "do we have a file?" It does not answer "does Lion sit weird next to a soft Mom?" or "is this Nepali letter actually usable?"

We kept regenerating from the terminal and losing the thread. So we built Lesson Studio: a tiny local web UI over the same bake library.

Filter by language, gender, and status (missing, ready, tagged, recorded, no-image). Search a word. Play it. Edit the speak prompt and regenerate one take. Record a mic override when the model will not get there. Check the lesson image, regenerate it, or upload a real one. Arrow keys for speed. Regenerate the asset map only when the batch feels done.

Mic overrides land on the same hashed path as a normal bake and get marked source: recorded in the manifest. The app key does not care how the bytes got there.

This is not a CMS in the cloud. It writes straight into assets/ and the manifest the app already understands. That constraint kept the tool honest.

Putting clips into the study screen

With clips shipping, study had to feel listen-first, not like a flashcard with nicer audio. We restyled it as a picture-book Listening Stage in the same release: tap art to hear, optional loop, autoplay countdown. The parent profile picks female or male voice. English lessons play English clips; Nepali alphabets, vowels, and numbers play Nepali. Dictation and spelling queue word → letters → word.

On device the happy path is simple: look up lang|gender|text, load the bundled Metro asset, play it with expo-audio. If a clip is missing, we still fall back to on-device speech. Kids almost never hit that path now.

Then the files started failing in app-shaped ways

Getting MP3s into git was the easy half. The week after is what we actually remember.

Spelling queues raced a single audio player. Swapping the source mid-queue (replace()) dropped letters or the final word, so A P P L E came out wrong even when every MP3 was fine. Fix: create a fresh player per clip, and share one small queue helper between study and games.

Alphabet names needed slower, clearer generation settings than a word like Lion. We rebaked A–Z and pruned the orphan letter files from the first pass.

Background lesson music still paused or ducked Spotify until we set the session to mix with other apps (interruptionMode: "mixWithOthers", with silent-mode playback allowed). Parents leave music on. The app should not police that.

Music kept looping after leaving a lesson because the background player lived in React state and cleanup sometimes held a stale reference. Moving it to a ref and always stopping on teardown ended the ghost loops.

Separately, TestFlight builds on an iOS beta host crashed about half a second after launch if expo-audio initialized too early (a native TurboModule abort). We moved audio (and a few other heavy natives) off the boot path: dynamic import on first use, defer analytics and purchases, and temporarily turn off React Native's New Architecture on that cursed host. Not content work. Still the difference between "we shipped clips" and "nobody can open the app to hear them."

And yes, we switched the repo to pnpm on the same branch. pnpm's default isolated node_modules layout broke local EAS iOS builds resolving react-native-css-interop. Hoisting installs fixed Metro. Unrelated to TTS. Extremely related to shipping an IPA that contains 1,700 MP3s.

None of that shows up in a green bake log.

What we would not undo

Generate offline. Review on your machine. Ship the files in the app. Streaming a voice API at tap-time is the wrong reliability model for toddlers.

Keep the lookup key separate from the speak prompt. Treat TTS providers as different dialects. Prune after every rebake. Build the review UI before you declare the bake finished.

On-device TTS is still the safety net. It is no longer what kids hear when they tap the apple.

FAQ