--- license: cc-by-4.0 language: - bg - hr - cs - da - nl - en - et - fi - fr - de - el - hu - it - lv - lt - mt - pl - pt - ro - ru - sk - sl - es - sv - uk library_name: coreml tags: - coreml - apple-silicon - ane - automatic-speech-recognition - canary - attention-encoder-decoder - aed - fluidaudio - speech-translation base_model: nvidia/canary-1b-v2 base_model_relation: quantized pipeline_tag: automatic-speech-recognition --- # Canary-1B-v2 — CoreML (ANE) CoreML conversion of [`nvidia/canary-1b-v2`](https://huggingface.co/nvidia/canary-1b-v2) for Apple Silicon / Neural Engine, packaged for [FluidAudio](https://github.com/FluidInference/FluidAudio). Canary is a **FastConformer encoder + Transformer attention encoder-decoder (AED)** ASR model (25 European languages, 16384-token SentencePiece BPE). It is decoded autoregressively: the transformer decoder cross-attends to the encoder output and emits tokens greedily until EOS (id 3), with a 1024→16384 projection head. ## Files | File | Role | Precision | |------|------|-----------| | `Preprocessor.mlmodelc` | waveform `[1,240000]` → mel `[1,128,1501]` | fp32 (CPU) | | `EncoderInt4.mlmodelc` | mel → encoder `[1,1024,188]` | int4 (ANE, iOS18) | | `DecoderInt4.mlmodelc` | autoregressive transformer → hidden `[1,256,1024]` | int4 (ANE, iOS18) | | `Projection.mlmodelc` | hidden `[1,1024]` → logits `[1,16384]` | fp16 (ANE) | | `vocab.json` | 16384 SentencePiece pieces (`id → piece`) | — | | `projection_weights.npz` | raw projection weights (for Python reference pipelines) | fp32 | | `metadata.json` | shapes, sample rate, special token ids | — | **Contract:** 15 s window (240000 samples @ 16 kHz), 256 decoder steps, `eos=3`, `pad=2`, `bos=4`. int4 weight payloads require **iOS 18 / macOS 15**. ## Variants - **int4** (this default): ANE-runnable, ~573 MB, fastest. Per-block-32 symmetric. - **fp16**: exact parity with PyTorch, iOS 17, ~1.8 GB (not included here by default). - int8 per-channel decodes correctly only on CPU (crashes the GPU/ANE MPSGraph backend), so it is not recommended; use int4 for an ANE-resident small build. ## Accuracy / speed (LibriSpeech test-clean, ≤15 s, int4, M-series ANE) | Metric | Value | |--------|-------| | WER | ~2.1% | | RTFx | ~7x | fp16 CoreML output is byte-identical to the NeMo PyTorch greedy decode. ## Usage (FluidAudio) ```swift let manager = try await CanaryManager.load(precision: .int4) let text = try await manager.transcribe(audioURL: url) ``` ### Speech translation (en ↔ 24 languages) The AED decoder picks its task from the two language slots in the prompt: `source == target` transcribes, `source != target` translates. The same model files translate speech with no change — only one prompt token differs. canary-1b-v2 is trained on en ↔ X pairs only. ```swift let translator = try await CanaryManager.load(source: .english, target: .german) let german = try await translator.transcribe(audioURL: url) // Audio longer than 15 s: split at pauses (Silero VAD) instead of overlap // chunking — translated text is not token-stable across an overlap, so the // seam merge used for transcripts degrades translations. let vad = try await VadManager() let longGerman = try await translator.transcribe(audioURL: longURL, vad: vad) ``` ```bash swift run fluidaudiocli canary-transcribe audio.wav --translate-to de [--source-lang en] [--vad] ``` Prompt token map, FLEURS benchmark hypotheses and scores: [FluidInference/canary-speech-translation-coreml](https://huggingface.co/FluidInference/canary-speech-translation-coreml). ## Conversion See the [mobius](https://github.com/FluidInference/mobius) conversion pipeline (`models/stt/canary-1b-v2/coreml/`): `convert-coreml.py` (NeMo→CoreML), `quantize_int4.py`, `build_projection.py`, `validate.py`, `stage_hf.py`. ## License Inherits `cc-by-4.0` from the base model `nvidia/canary-1b-v2`.