Generic TTS engines speak English, sometimes Afrikaans, badly. For a non-verbal child whose home language is Afrikaans, "badly" isn't acceptable — it's the difference between communication and frustration.
We trained a custom Afrikaans voice for an AAC app from a single-speaker corpus of 2,731 phrase-level samples. The model runs on-device on iOS and Android, with no network call. Here's the engineering — dataset prep, model choice, training run, on-device deployment, and the things we'd do differently next time.
Dataset prep: the 60% of the work
We recorded 2,731 phrase-level samples in a controlled studio: single speaker, single mic, consistent room tone, 22.05kHz mono. That's the easy part. Phonetic balance is harder — Afrikaans has phonemes (rolled R, gutturals like /x/) that are underrepresented in English-derived training corpora and need explicit dataset bias to come out clean.
Forced alignment with the Montreal Forced Aligner gave us phoneme-level timestamps. We hand-cleaned about 12% of samples — usually mic clipping, breath noise, or alignment errors where the speaker dropped a phoneme. Our cleanup pipeline rejected another ~6% outright.
If we ran this again we'd build a more aggressive dataset curation tool upfront — automated rejection on signal-to-noise ratio, on speaker-consistency drift across sessions, and on phoneme balance against a target distribution. Cleanup at training time is the slowest possible loop.
Why VITS over Tacotron 2 / FastSpeech 2
Tacotron 2 + WaveGlow gave decent quality but required two separate models in production — slow load, more memory, more code surface to break. FastSpeech 2 was fast but the prosody on Afrikaans interrogatives sounded mechanical.
VITS is end-to-end (text → waveform, no separate vocoder) and the variational latent space gave us natural prosody variation without explicit prosody labels. Smaller deployable size, fewer moving parts, better Afrikaans-specific quality after equivalent training time. It became the obvious choice.
Training run #14
Training runs 1 through 13 had specific issues: run 7 collapsed to a single voice mode, run 9 had inconsistent volume across phrase types, run 11 sounded robotic on long phrases (we'd over-regularised the duration predictor). We kept a training log noting hyperparameters and qualitative outcomes.
Run 14 — 800k steps on a single A100, ~5 days of compute — was the one that didn't sound robotic. Hindsight: we should have started with the published reference hyperparameters and only tuned after the first viable model, not before. The early tuning was educated guessing without a baseline to compare against.
ONNX export and on-device deployment
VITS exports cleanly to ONNX with one caveat: the variable-length output tensor needs explicit dynamic axes. The custom-op pattern for the flow-based decoder needs care during export — in our case we patched the export script to avoid an unsupported op and reroute it through a supported equivalent.
The deployable model is ~38MB after quantization. Inference latency for a typical 5-word phrase: ~280ms on iPhone 11+, ~600ms on a Samsung A52. For an AAC app, "instant" is anything under ~500ms — we hit it on flagship phones and felt acceptable on mid-range Android.
What we'd do differently
More samples. 5,000+ would have closed prosody gaps on long sentences without needing a reference encoder.
Multi-speaker training. Even if you ship one voice, multi-speaker training acts as a regulariser and gives better prosody. Single-speaker fine-tunes on top.
Explicit prosody control via reference encoder. Useful for emphasis and emotion, especially for AAC where emotional tone is part of the message.
Build the dataset curation pipeline before recording, not after. We'd save weeks of cleanup.
Custom voice models for low-resource languages are no longer the preserve of research labs. With ~2–3k high-quality samples, VITS, and a few weeks of compute, you can ship a usable on-device voice for any language you have native speakers and recording capacity for.
The hardest part isn't the model — it's still dataset curation, and it's still the unglamorous part of the work. If you're considering a custom voice model and you can't answer "who does the recordings, where, and how do you reject bad samples?" before you start, start there.
