This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
What I Built
tonesearch searches call recordings by what was said and how it was said.
You type "customer wants a refund" and pick the tone "angry". You get back the 20 audio segments that match both. Click a row and you hear the segment. A second tab shows the tone over one whole call, with the top 3 spikes.
It needs no transcript and runs on a laptop, so no audio leaves the machine.
The people I think would pay are small call teams and small agencies that run voice agents for clients. A voice agent can make hundreds of calls a day. Somebody has to find the calls where the caller got angry, and fix the prompt. Today that person listens to random calls, or reads transcripts that lose the tone.
Big contact centers buy tools like Gong or Cresta for this. Those tools are built for large teams, and a five-person agency is not their customer. Some agencies also cannot upload customer calls to another company's cloud, because their clients forbid it.
What they want is small: "show me the angry moments in this week's calls, and don't send the calls anywhere."
Demo
Try it: https://huggingface.co/spaces/pradachan/tonesearch
The demo runs in your browser. The Gemma query model (about 142 MB, q4 ONNX through transformers.js) downloads once, then your query is embedded on your own machine. Tone-only search works while it loads. The query embedding in the browser matches the Python one at cosine 0.989.
Search tab. The query is "customer wants a refund", tone "angry", tone weight 1.
Timeline tab. The angry score over one call, with the top 3 spikes.
The demo data is HaessigDB, a public set of acted call-center speech (CC-BY-4.0, by Niklas Weller, Marc Christopher Grau and Ivo Blohm). These are actors, not real customers.
Code
tonesearch
Search call recordings by what was said and how it was said. Runs locally. No transcript needed.
- What: EmbeddingGemma 2 embeds the audio, so a text query finds speech by meaning.
- How: emotion2vec+ scores each segment for tone (angry, sad, happy, ...).
- Timeline: tone over time for one call, with the top 3 spikes.
Validation numbers: RESULTS.md.
Where
| File | What |
|---|---|
tonesearch.py | Indexer, search, CLI, Gradio UI |
make_demo.py | Builds whole-call demo WAVs from HaessigDB |
test_tonesearch.py | Self-check on the demo index |
eval_tone.py, probe_tone.py, probe_e2v.py | Research scripts behind RESULTS.md |
How
pip install -e . tonesearch index <audio dir> <index dir> # any .wav .mp3 .flac .m4a .ogg, resampled to 16 kHz mono tonesearch search <index dir> "refund request" --tone angry --w 1 tonesearch search <index dir> --tone angry # tone only tonesearch ui <index dir> # Gradio… About 170 lines of Python in one file, plus a self-check:
pip install -e . tonesearch index <audio dir> <index dir> tonesearch search <index dir> "refund request" --tone angry --w 1 tonesearch ui <index dir> How I Built It
Two open-weight models do the work.
- EmbeddingGemma 2 (
google/embeddinggemma-2) embeds each audio segment. It also embeds the text query, in the same space. So a text query finds speech by meaning, with no speech-to-text step. - emotion2vec+ base (
emotion2vec/emotion2vec_plus_base) scores each segment for tone labels such as angry, sad, happy and neutral.
The indexer splits each call on silence into 2 to 10 second segments. It runs both models once and saves one .npz file. A search is one matrix multiply:
score = z(cosine to query) + w * z(tone score) w = 0 searches by words only. An empty query searches by tone only.
Gemma alone did not hear the tone
My first plan was simpler. EmbeddingGemma 2 takes audio in, so I assumed it would hear an angry voice too. Embed the calls, search for "angry customer", done.
I'm glad I tested that before building anything. HaessigDB has 888 clips, each rated 1 to 10 for irritation. I measured how well each signal tracks that rating. I also measured what is left after a transcript model explains what it can. I call that column "beyond words". For the trained probes, the test voice was never in the training set.
| Signal | Spearman | Beyond words |
|---|---|---|
| EmbeddingGemma 2 audio, query "angry customer" | +0.36 | +0.11 |
| EmbeddingGemma 2 audio embedding, trained probe | +0.45 | +0.06 |
| EmbeddingGemma 2 transcript embedding, trained probe | +0.45 | n/a |
| Loudness | +0.42 | +0.41 |
| emotion2vec+ "angry" score, no training | +0.54 | +0.54 |
| emotion2vec+ and EmbeddingGemma 2 audio, trained probe | +0.66 | +0.45 |
The audio embedding scored the same as the transcript embedding. It hears the words about as well as a transcript does, and it adds almost nothing about tone. Plain loudness, which is one line of librosa, did better on tone than Gemma did. That surprised me.
emotion2vec+ caught tone with no training at all. Inside one call, its angry score follows the escalation (median Spearman +0.41 over 37 calls).
So each model got one job: Gemma handles what was said, and emotion2vec handles how it was said.
One more fix: ties at the top
emotion2vec returns softmax scores. On the 60-minute demo, 133 of 1176 segments scored exactly 1.0 for "angry". The top of an angry search was a tie.
The fix was small. I now rank on log-odds: the log score minus the mean log score over all labels. That is the model's own logit gap, and it does not saturate.
| Tone-only "angry" search, 1176 demo segments | Spearman | Mean irritation, top 10 |
|---|---|---|
| Softmax score | +0.50 | 7.19 |
| Log-odds | +0.52 | 7.54 |
Speed
On an M3 Mac, with Gemma on the GPU, indexing runs at 18x real time. On CPU only, it runs at 3.4x real time. Gemma is the slow part on CPU. An int8 ONNX build of the Gemma audio encoder is my next step.
What it does not do yet
- The test data is acted and scripted. Real calls are noisier.
- HaessigDB has only 4 actors.
- It has no transcript and does not split speakers. You hear each segment, but you cannot read it.
Why Does Open Innovation Matter?
Without open weights I could not sell this to the people above.
The first reason is privacy. Call audio is full of names, account numbers and sometimes health details. Open weights let the models run where the audio already lives, so nothing gets uploaded. For a lot of small agencies that is the only way a tool like this is allowed at all.
The second is cost. There is no per-minute API fee. An M3 laptop CPU indexes a 5-minute call in about 90 seconds, so the cost per call is close to zero, and a small flat monthly price could still leave a margin. I have not picked a price yet.
The third is that I could look inside. I pulled the embeddings out and trained probes on them, which is how I found that Gemma audio hears words and not tone. A closed API would have given me a score and nothing to check it against.
The last is that the parts are swappable. When one model could not hear tone, I added a second one. When a better tone model comes out, I change one model name, and no vendor has to sign off.
A license caveat: emotion2vec+ uses the FunASR model license. I must check its commercial terms before I charge anyone.
Prize Categories
- Best Use of Gemma: EmbeddingGemma 2 is the search engine. It embeds call audio and text queries in one space, so you can search speech by meaning with no transcript.
