parakeet-tdt-0.6b-unconstrained

nvidia/parakeet-tdt-0.6b-v3 fine-tuned for the Speech Accessibility Project Challenge 2 unconstrained track, which permits training data from outside the challenge corpus. On top of the Speech Accessibility Project recordings this run adds two further dysarthric corpora -- HeyJay! (ICPSR 39448) and a Project Relate takeout of UK ataxic speech -- with online speed perturbation and SpecAugment. A token-and-duration transducer, not CTC: it has a prediction network, a joint network and a duration head, and decodes autoregressively.

It is the best model in this family, by a small margin. All rows are the same 17,582 utterances from 48 speakers, no language model, both sides through the same normaliser:

	WER	CER
nvidia/parakeet-ctc-0.6b	32.35%	20.43%
nvidia/parakeet-tdt-0.6b-v3	29.49%	19.99%
dys-asr/parakeet-ctc-0.6b-sapc1	14.65%	8.77%
dys-asr/parakeet-ctc-0.6b-sapc2	13.35%	7.83%
dys-asr/parakeet-ctc-0.6b-all	12.08%	7.02%
dys-asr/parakeet-ctc-0.6b-soup	11.89%	6.89%
dys-asr/parakeet-ctc-0.6b-all-aug	11.65%	6.80%
dys-asr/parakeet-tdt-0.6b-all-aug	10.73%	6.67%
this model	10.52%	6.45%

Be honest about what the extra data bought. parakeet-tdt-0.6b-all-aug is this model's controlled comparison -- same architecture, same recipe, same seed, differing only in that it lacks HeyJay! and Project Relate. The gap is 0.21 WER points, from 15.5 hours of added speech against 905 hours of Speech Accessibility Project audio. On the 126,993 reference words of the held-out set the standard error of one such WER is about 0.09 points, so 0.21 is a little over two of them unpaired -- and word errors are correlated within utterances, which widens the true interval. Treat the direction as likely and the magnitude as unresolved. The reason to prefer this checkpoint is that it is not worse and it is what the unconstrained track allows, not that 15.5 hours transformed the model.

Requirements

transformers>=5.9. ParakeetForTDT does not exist before it, so the 4.x releases that run the CTC siblings cannot load this model.

import torch
from transformers import AutoProcessor, ParakeetForTDT

model_id = "dys-asr/parakeet-tdt-0.6b-unconstrained"
processor = AutoProcessor.from_pretrained(model_id)
model = ParakeetForTDT.from_pretrained(model_id).eval()

inputs = processor(audio, sampling_rate=16_000, return_tensors="pt")  # 16 kHz mono
with torch.inference_mode():
    outputs = model.generate(**inputs)
print(processor.batch_decode(outputs.sequences, skip_special_tokens=True))

Decoding is autoregressive rather than an argmax over frames, so it is substantially slower than the CTC siblings -- roughly an order of magnitude fewer samples per second.

Output text convention

Numbers as words, lower-case, unpunctuated.

audio:  "lower the temperature three degrees"
output: lower the temperature three degrees     # not "... 3 degrees"

Lower case is not cosmetic. v3's tokeniser holds 8,192 tokens and is overwhelmingly lower-case -- only 740 contain any capital -- so upper-case labels cost 4.06 tokens per word against 1.67, words shattering into single characters. Labels are lower-cased before tokenising; scoring is unaffected, since the normaliser upper-cases both sides. Note this differs from the CTC siblings, which emit upper case.

Read the evaluation set carefully

Training includes SAPC1 dev, which shares 28,253 recordings and 76 of SAPC2 dev's 124 speakers, so neither sapc1_dev nor the whole of sapc2_dev is a fair test here. The set above is the 48 SAPC2 dev speakers appearing nowhere in training, 17,582 of 47,929 records.

It is harder than SAPC2 dev as a whole: all 35 of SAPC2 dev's Parkinson's speakers also appear in SAPC1 dev, so it contains no Parkinson's speech, easily the lowest-error cohort. Do not compare this model's 10.52% against parakeet-ctc-0.6b-sapc2's published 11.51% on all of SAPC2 dev.

This model has no out-of-corpus test set left. HeyJay! and the Project Relate takeout are here in full -- all 8,577 and all 1,351 recordings -- so the cross-corpus scores published for dys-asr/parakeet-tdt-0.6b-sapc1 (11.93% WER on HeyJay!, 22.81% on the takeout, both zero-shot) say nothing about this checkpoint, and rerunning them here would be scoring the training set. That is a deliberate trade for the unconstrained track, not an oversight.

Training data

Deduplicated by audio filename, filtered to 0.5-30 s and to labels of at most 130 tokens: 402,130 utterances of 413,442 candidates.

corpus	recordings	duration	speakers
SAPC1 train + dev	250,014	573.5 h	663
SAPC2 train	153,500	331.1 h	460
HeyJay! (ICPSR 39448)	8,577	13.3 h	38
Project Relate takeout, UK ataxia	1,351	2.2 h	1
total	413,442	920.1 h	1,094

The speaker total is below the column's sum because 68 contributors appear in both SAPC1 and SAPC2. The two added corpora are 2.4% of records and 1.7% of duration, which is the context for the 0.21-point margin above.

SAPC2 re-releases 182,575 of SAPC1's train recordings under the same filenames, so concatenating the manifests would train on them twice. Square-bracketed spans, which in SAPC2 hold the interview prompt shown to the speaker rather than read aloud, are stripped before normalisation.

The Speech Accessibility Project data is governed by its own data use agreement, and HeyJay! (ICPSR 39448) and the Project Relate takeout each carry separate terms. Redistributing the weights is not redistribution of any of those corpora, but anyone reproducing this run needs access to all three.

Augmentation

Both applied online in the collator, so the batch size never changes:

• Speed perturbation at 0.9x, 1.0x or 1.1x, one factor per utterance, by rational resampling with the rate left declared as it was.
• SpecAugment, masking time and frequency spans of the mel features. The span sampler is borrowed from Transformers' Wav2Vec2; the Parakeet port has none.

Speed runs before feature extraction so the frame count and attention mask follow the perturbed audio, and the time mask honours that attention mask so spans land inside the utterance rather than in padding.

Training hyperparameters

base model	nvidia/parakeet-tdt-0.6b-v3
optimizer	AdamW, weight decay 0.01
learning rate	1e-4
schedule	tri-stage, 10% warmup, 40% hold
per-device batch	8 x 2 GPUs, SyncBatchNorm over 16
gradient accumulation	2
effective batch	32
epochs	10 (125,670 optimizer updates)
precision	bf16 mixed, float32 weights
layerdrop	0.05
gradient clipping	1.0
max audio duration	0.5-30 s
max label length	130 tokens
gradient checkpointing	off
batch sampling	length-grouped
speed perturbation	0.9 / 1.0 / 1.1
SpecAugment	time 0.05 x 10 frames, frequency 0.4 x 27 bins
seed	42
hardware	2x NVIDIA A100-SXM4-80GB

The learning rate is the winner of a four-point sweep run under this exact recipe, one epoch each, scored on the same 4,000-utterance subset: 3e-5 gave 11.17% WER, 5e-5 gave 10.51%, 1e-4 gave 10.07%, 2e-4 gave 10.35%.

Weights stay float32 under bf16 autocast: loading in bfloat16 fails outright, because the feature extractor emits float32 and the first convolution rejects the mismatch.

Dev WER on a fixed 4,000-utterance subset, by epoch:

epoch	1	2	3	4	5	6	7	8	9	10
WER	13.66	11.98	11.41	11.46	11.08	11.03	10.43	10.80	10.18	9.94
CER	8.87	7.73	7.23	7.29	6.91	6.76	6.44	6.99	6.30	6.05

Epoch 10 is the best epoch and the released weights are its. That subset is easier than the full held-out set -- 9.94% there against 10.52% over all 17,582 utterances -- so the two columns are not interchangeable. Unlike parakeet-tdt-0.6b-all-aug, which turned at epoch 9, this run was still improving when the schedule ended, so more epochs may still pay.

Limitations

• Requires transformers>=5.9. Not loadable by the 4.x line.
• Slow relative to CTC. Autoregressive decoding.
• Lower-case, unpunctuated, numerals as words -- and lower case differs from the CTC siblings, which emit upper case.
• Multilingual base fine-tuned on English only, discarding capability it had.
• No out-of-corpus evaluation is possible, since HeyJay! and the Project Relate takeout are entirely in training.
• No fair comparison against the sibling models on their own test sets, since this trains on SAPC1 dev.
• The held-out set excludes Parkinson's speech entirely.
• The UK ataxia data is a single speaker, so it adds an accent and an etiology rather than speaker diversity.
• BatchNorm statistics span 16, not 32, unlike the CTC models.
• No per-speaker breakdown, since the final evaluation did not retain per-utterance predictions.
• Single training seed. No variance estimate.