Give Echo a few seconds of someone's voice and some text, and it reads the text back in that voice. A 3D avatar on screen moves its lips in sync with the audio. It's all self-hosted and runs on CPU: no GPU, no paid voice API, and the voice samples never leave your machine.
Echo started in June as a weekend experiment. It ran once, on my laptop. Over the past few weeks I turned it into a small voice studio. It can now write the script for you, keep a library of voices and a history of everything you generated, and let you tune how the voice sounds.
But Echo isn't the end goal. I built it to learn how text-to-speech really works, so I can bring it into my RAG project and, later, build a Memorial App. More on that below, then a tour of Echo and the lessons I picked up along the way.
The code is on GitHub: jbcanillo/echo.
Why I'm building this
I already run a RAG chatbot, Mnemosyne, that answers questions from a knowledge base. The next step I want to take with it is personal: a Memorial App.
The idea: while someone is still alive, they prepare what they want to leave behind. That could be messages for their family, their will and testament, stories and memories, or answers to the questions their kids might ask someday. After they pass, their loved ones can hear those messages in that person's own voice, and ask questions that are answered from what they left.
To get there I need to understand TTS properly, not just call an API. How does a model clone a voice from a few seconds of audio? What do its settings actually change? Can it run on modest hardware, without sending anyone's voice to a third party? Echo is where I'm learning those answers hands-on.
It's also practice for the part that matters most: security and consent. A person's voice, their last wishes and their family's grief are about as sensitive as data gets. A Memorial App has to guarantee that:
- The voice is cloned only with the owner's own consent, given while they're alive.
- Messages, wills and memories stay private, and only the people they chose can reach them.
- Families can always tell apart what their loved one actually recorded and what the AI generated from it.
That's why the safeguards in Echo aren't an afterthought. They're the first draft of the ones the Memorial App will need.
A quick tour
Echo is a mobile-first PWA built around two steps: pick a voice, then make it say something.
| Tab | What you do there |
|---|---|
| Sample | Record your voice (with a live level meter), upload a clip, or pick one from your library |
| Synthesize | Type a script, or ask AI to write one ("a 5 paragraph bedtime story about friendly monsters"), then generate |
| Library | Saved voices you can reuse without re-recording |
| History | Your last 50 generations, with replay, download and "reuse text" |
| Settings | OpenRouter key and model, plus voice tuning: speed, expressiveness, variation and language |
While you record, the avatar listens. While the backend works, it strikes a thinking pose. When the audio plays, its lips follow it. You can switch between a female and a male avatar.
Under the hood there are three parts, started with one docker compose up:
- Frontend: React + Vite, served by nginx, with the avatar rendered in the browser.
- API: a Node/Express server that validates requests, runs the AI text endpoints and manages the voice jobs. It also serves Swagger docs at
/api/docs. - Worker: a Python process running Coqui TTS with the YourTTS multilingual model (English, French and Brazilian Portuguese).
![How Echo fits together: browser, API, Python worker, optional OpenRouter]

All configuration lives in one .env file at the repo root, shared by Docker Compose, the backend and the frontend. Only VITE_* variables reach the browser, so the OpenRouter key never ends up in the JavaScript bundle.
Lesson 1: Load the model once, then tune it per request
The June version did what most "call Python from Node" tutorials show. Each request spawned a new Python process, which imported Coqui and loaded YourTTS from scratch:
// June: one Python process per request app.post('/api/clone-voice', upload.single('voiceSample'), (req, res) => { const pythonProcess = spawn('python3', [pythonScript]); pythonProcess.stdin.write(JSON.stringify(inputData)); // ...collect stdout, parse JSON when the process closes }); Loading the model is the slowest part of the job, so every request paid for it again. Two requests at once meant two copies of the model in RAM.
One worker, JSON lines
Now one Python process stays alive. Node sends one job per line on stdin, and the worker answers with one result per line on stdout. The model loads on the first job and stays in memory.
# voice_clone.py protocol_out = sys.stdout sys.stdout = sys.stderr # TTS prints progress; keep stdout clean for JSON for line in sys.stdin: job = json.loads(line) result = process(job) # model is cached after the first call result["id"] = job.get("id") protocol_out.write(json.dumps(result) + "\n") protocol_out.flush() That sys.stdout = sys.stderr swap matters. Coqui and PyTorch print progress bars to stdout. If they share the pipe with your JSON, Node gets garbage halfway through a result.
On the Node side, a small queue runs jobs one at a time, matches results by id, and kills and restarts the worker if a job runs past JOB_TIMEOUT_SECONDS (600 by default, enough for the first-run model download of about 425 MB). On a CPU, two syntheses in parallel would each run at half speed and use twice the memory, so a queue gets about the same work done without the risk.
Tuning a model that everyone shares
The newest feature lets you set speed, expressiveness and variation per request. YourTTS is a VITS model, so these map onto three attributes of the loaded model:
| Echo setting | VITS attribute | Range |
|---|---|---|
| Speed | length_scale (set to 1 / speed) | 0.5 to 2 |
| Expressiveness | inference_noise_scale | 0.1 to 1 |
| Variation | inference_noise_scale_dp | 0 to 1.5 |
Here's the catch: the model is shared. If one request sets speed to 2 and doesn't put it back, every request after it talks fast too. So the worker saves the old values, applies the new ones, and restores them in a finally block, even if synthesis throws:
previous = {} try: tts = get_tts() previous = apply_synth_settings(tts, job) # returns the values it replaced tts.tts_to_file(text=text, speaker_wav=sample, language=lang, file_path=out) finally: model = tts.synthesizer.tts_model for attr, value in previous.items(): setattr(model, attr, value) The API also checks every value against its range before a job reaches the queue, so a bad request gets a clear 400 error instead of a strange-sounding voice.
Lesson 2: Docker on CPU has its own traps
I wanted Echo to run on ordinary hardware, so the backend image is Node 20 plus Python plus ffmpeg, with no CUDA. These are the things that bit me.
1. The default PyTorch is the CUDA build. That's gigabytes of GPU libraries you'll never use. Coqui (I use the maintained coqui-tts fork) doesn't install PyTorch for you anyway, so you install it yourself from PyTorch's CPU index.
2. --index-url vs --extra-index-url. My first Dockerfile used --index-url for the CPU wheels. That index only hosts torch packages, not their dependencies like numpy or typing_extensions, which can break the install. Keep PyPI as the main index and add PyTorch's as an extra. Install everything in one pip call, so the resolver picks torch versions that coqui-tts accepts too:
RUN python3 -m venv /opt/venv ENV PATH="/opt/venv/bin:$PATH" RUN pip install --no-cache-dir --extra-index-url https://download.pytorch.org/whl/cpu \ torch torchaudio torchcodec -r requirements.txt 3. Undeclared dependencies. With PyTorch 2.9 and up, audio loading needs torchcodec. One of Coqui's dependencies also imports six without declaring it. Neither is in Coqui's requirements, so both are listed explicitly.
4. One missing underscore broke every request. My default model name was .../multi-dataset/yourtts. The real ID is your_tts. Coqui threw a KeyError inside the worker, and the API just said "Voice cloning failed". If you hit KeyError: 'yourtts', that's why.
5. nginx doesn't know .mjs. The avatar's lip-sync runs in an AudioWorklet loaded from an .mjs file. Browsers refuse to load a module that isn't served as JavaScript, and nginx's default MIME types don't include .mjs. Two lines in nginx.conf fix it:
include /etc/nginx/mime.types; types { text/javascript mjs; } One more tip: mount a volume for the model cache (TTS_HOME=/app/models). Otherwise every rebuild downloads the 425 MB model again.
Giving the voice a face
The avatar uses two libraries by met4citizen:
- TalkingHead renders a rigged GLB avatar with Three.js and handles idle motion, gestures and eye contact.
- HeadAudio listens to an audio stream and predicts mouth shapes (visemes) from the sound itself, in an AudioWorklet.
TalkingHead normally lip-syncs from text. But Echo's audio comes from my backend, and the cloned voice won't match text-based timing. Driving the lips from the audio means they follow whatever actually plays.
The wiring is a few Web Audio nodes. The player feeds the avatar's speech gain node, which feeds HeadAudio (for the lips) and, through a short delay, the speakers:
head.audioSpeechGainNode.connect(headAudio); // HeadAudio needs ~100 ms to detect a viseme, so delay what you hear to match const delay = new DelayNode(audioCtx, { delayTime: 0.1 }); head.audioSpeechGainNode.disconnect(head.audioReverbNode); head.audioSpeechGainNode.connect(delay); delay.connect(head.audioReverbNode); That 100 ms delay is what makes it look right. Without it, the mouth moves a beat after each sound.
A fallback for plain HTTP
Self-hosted apps often get opened from another device on the LAN, over plain HTTP. There, precise lip-sync can't work: browsers only allow AudioWorklets in a secure context: HTTPS or localhost. On http://192.168..., audioCtx.audioWorklet is simply undefined.
Instead of failing, Echo now falls back to a simpler lip-sync. It measures the loudness of the audio every frame and opens the jaw to match, smoothed so it doesn't flutter:
const loudness = Math.min(1, rms(samples) * 4); level += (loudness - level) * 0.4; // smooth the jaw head.mtAvatar.jawOpen.newvalue = level * 0.7; head.mtAvatar.viseme_aa.newvalue = level * 0.5; It's not as accurate as real visemes, but a talking face beats a frozen one. Serve over HTTPS and you get the full version.
Swapping avatars
You can now switch between a female and a male avatar, and the model swaps without rebuilding the audio graph. One gotcha: if male.glb is missing, the dev server and nginx answer with index.html instead of a 404. So before loading, Echo fetches the first 4 bytes and checks for the GLB magic number, glTF. If it isn't there, it falls back to the default model.
Two more browser gotchas:
- An
<audio>element can only be wired into oneMediaElementAudioSourceNode, ever. React re-renders will try to create a second one and throw. I cache the source in aWeakMapkeyed by the element. - Browsers keep the
AudioContextsuspended until the user interacts with the page. I resume it on the firstpointerdownorkeydown.
Keeping your data on your device
A voice library and a generation history sound like they need a database on the server. They don't. Both live in the browser's IndexedDB, so saved voices and generated audio never sit on the server at all.
export async function addHistoryItem(item: HistoryItem): Promise<void> { await run('history', 'readwrite', (s) => s.put(item)); // Keep storage bounded: drop the oldest generations beyond the cap const all = await listHistory(); await Promise.all(all.slice(MAX_HISTORY).map((old) => deleteHistoryItem(old.id))); } IndexedDB stores Blobs directly, so a saved voice is just the original recording plus a name and a date. History keeps the latest 50 items, so a busy afternoon can't quietly fill the disk.
Every saved item gets a waveform player, which means decoding audio. Decoding 50 clips when the History tab opens would be slow, so each player waits for an IntersectionObserver and only decodes once it scrolls into view.
A settings page that never shows your key
You can now set the OpenRouter key and model from the app instead of editing .env. The key is write-only: the API never sends it back. The browser only learns whether a key exists, where it came from, and its last four characters:
function publicOpenRouter() { const key = getApiKey(); return { configured: Boolean(key), source: stored.openrouterApiKey ? 'settings' : process.env.OPENROUTER_API_KEY ? 'env' : 'none', keyHint: key ? `…${key.slice(-4)}` : '', model: getModel() }; } The saved settings go in a JSON file inside the same Docker volume as the models, so they survive rebuilds. The file is written to a temp file and renamed into place (so a crash can't leave half a file), with 0600 permissions. There's also a "Test key" button that checks the key against OpenRouter before you save it.
Consent and safety, on by default
A tool that copies someone's voice can be misused, so I treated safeguards as features, not an afterthought. All of these are on out of the box:
- Consent is checked twice. Generate stays disabled until you tick "This is my voice, or I have the speaker's permission to clone it", and the API rejects any request without
consent=true. Your script survives a page reload, but the consent box deliberately doesn't: you confirm it again each session. - The server keeps nothing. The uploaded sample, its converted copy and the generated WAV are deleted after every request, including failed and timed-out ones. Your library and history live only in your browser.
- Limits on everything. One audio file per request (20 MB max), 5,000 characters of script, 1,000 characters for an AI prompt, 5 requests per minute, and a timeout on every job. Most of them are settings in
.env. - Rate limits count real clients. Behind nginx, every request looks like it comes from the proxy. Setting Express's
trust proxyto exactly one hop makes the limiter count by the real client IP instead of putting everyone in one bucket. - Audio stays local. Only script text ever goes to OpenRouter, and only when you use Write text or Polish wording.
One honest caveat, which the Settings page also states: anyone who can open the app can replace the saved OpenRouter key. Keep Echo on a private network.
That caveat is fine for a learning project on my own network. It's exactly what a Memorial App can't have. There, the bar is much higher: real user accounts, access limited to the people the owner chose, and encrypted storage for every message and document. Echo taught me where those gaps are.
The cleanup was trickier than it looks. Files have to go whether the job succeeds, fails, times out, or the user closes the tab halfway. Hooking the cleanup to the response's close event covers all four:
res.on('close', () => { removeFile(inputPath); removeFile(processedPath); if (!responded) removeFile(outputPath); }); Wrapping up
Echo started as a one-off script and became a small studio: one docker compose up, a model that loads once, a face that talks, and safeguards that don't depend on anyone reading the docs. More importantly, it taught me how TTS works under the hood.
Next, I'll connect what I learned to Mnemosyne, so answers from a knowledge base can be spoken in a chosen voice. That's the foundation for the Memorial App: a secure place where someone can leave messages, their will and their memories for the people they love, in their own voice.
If you're calling a Python model from a Node (or PHP, or Go) app, the main takeaways are:
- Keep the model warm in one long-lived process and talk to it over JSON lines.
- Queue the jobs and put a timeout on everything.
- If requests can tune a shared model, restore its settings in a
finallyblock. - Plan a fallback for every browser feature that needs HTTPS.
The full source, with a detailed README, Swagger API docs and a troubleshooting guide, is at github.com/jbcanillo/echo. Questions and PRs are welcome. If you've worked on TTS, voice cloning, or apps that handle this kind of personal data, I'd love to hear your advice in the comments.