FranciscoCarlosErra/ttslibre-poc-es-en

TTSLibre – Spanish + English proof of concept (experiment 016)

Proof-of-concept checkpoint from TTSLibre, an open recipe for a small zero-shot TTS (Supertonic-lineage: character input, flow matching over a compressed autoencoder latent, Vocos vocoder). ~19.8M parameters, trained from scratch on an interleaved 50/50 mix of 4 h Argentinian Spanish (OpenSLR 61) and 4 h English (LibriTTS-R), fewest speakers per language. Same architecture and code as ttslibre-poc-en and ttslibre-poc-es-ar, plus language tags: every text is wrapped in <es>...</es> or <en>...</en> (4 single-token tags appended to the vocab, 80 -> 84). Not a usable product: intermediate checkpoint of a run still in progress, chosen by wall time, no whole-val evaluation (see the repo's experiment 016).

Files

file	what
ttl.pt	TTS model, experiment 016 checkpoint at 3h13m / step 54000 (weights only)
ae.pt	mel autoencoder (24-d latent, 6x compression), experiment 001, shared with the other PoCs
vocab.json, latent_stats.pt	character vocab (English + Spanish letters + 4 language tags) and latent normalization (same stats as the other PoCs)
config.json	effective training config of the run
ref_default.wav	default reference clip (OpenSLR 61, speaker arf_00295)
voices/*.pt	voicepacks: AE latents of a reference clip; arf_*/arm_* OpenSLR 61 speakers (in the training data), 4137/6209 LibriTTS speakers (not in the training data, zero-shot)

Vocoder: charactr/vocos-mel-24khz (loaded from the Hub at runtime).

Training

4.0 h OpenSLR 61 es-AR + 4.0 h LibriTTS-R English, rows alternating es/en, 5% held out, 1x RTX 3090, batch 32-48, lr 5e-4, random init. Probe WER during training (Whisper, 1 draw per sentence per round, Spanish reference voice for both languages; heldout = 3 val sentences per language, novel = hand-written):

elapsed	step	heldout en	novel en	heldout es	novel es
2h24m	42000	0.73	0.00	0.50	0.67
2h36m	45000	0.54	0.29	0.72	0.56
2h49m	48000	0.47	0.00	0.46	0.44
3h01m	51000	0.40	0.14	0.79	0.56
3h13m this file	54000	0.63	0.14	0.38	0.78

Reference: the Spanish-only PoC (014, 8 h) had held-out 0.38 at 2h21 and 0.10 at 3h30.

Inference from a cold start

Tested with Python 3.12, torch 2.10 (CUDA 12.8), GPU or CPU. The language is detected from the text (Spanish characters / stopwords; default English) and tagged automatically; pass lang="es"/"en" to force it.

git clone https://github.com/franciscocarloserra/ttslibre && cd ttslibre
python3 -m venv venv && . venv/bin/activate
pip install torch torchaudio vocos soundfile numpy num2words einops huggingface_hub
hf download FranciscoCarlosErra/ttslibre-poc-es-en --local-dir hfckpt
E=experiments/20-scaling/016-mixed-es-en
mkdir -p $E/runs/esen $E/runs/ae datasets/mixed-es-en/prep datasets/openslr61
cp hfckpt/ttl.pt $E/runs/esen/ttl.pt
cp hfckpt/ae.pt $E/runs/ae/ae.pt
cp hfckpt/latent_stats.pt hfckpt/vocab.json datasets/mixed-es-en/prep/
cp hfckpt/ref_default.wav datasets/openslr61/arf_00295_00000740990.wav
cp hfckpt/voices/*.pt experiments/voices/
cd $E
python synth.py "Mañana vamos a la playa con los chicos." hola.wav --run runs/esen/ttl.pt --ref ../../voices/arm_06136.pt
python synth.py "The quick brown fox jumps over the lazy dog." fox.wav --run runs/esen/ttl.pt --ref ../../voices/arf_06136.pt
# add --cpu for CPU; --ref can also be any 5-10 s wav of a new speaker

In-memory use (load once, generate many)

import sys, soundfile as sf
sys.path.insert(0, "experiments/20-scaling/016-mixed-es-en")
from common import load_config
from synth import Synth
c = load_config("experiments/20-scaling/016-mixed-es-en/config.json")
S = Synth(c, run="runs/esen/ttl.pt")
zref, rmask = S.style("../../voices/arf_06136.pt")   # voicepack, or a wav path for a new speaker
for i, t in enumerate(["Primera oración.", "Second sentence."]):
    wav, dur = S(t, zref, rmask, steps=16, cfg=2.0, duration_scale=1.0)   # lang="es" / "en" to force
    sf.write(f"out{i}.wav", wav, c["data"]["sample_rate"])

License

Weights and voicepacks CC BY-SA 4.0: training data is OpenSLR 61 (Google, CC BY-SA 4.0, https://www.openslr.org/61/) and LibriTTS-R (CC BY 4.0). The audit is in the repo's docs/LICENSES.md. Code MIT.