Automatic Speech Recognition

Everything to know about audio-to-text: how ASR works, the mid-2026 model landscape, evaluation metrics, and runnable code to test the leading open models.

1. What is ASR?

Automatic Speech Recognition (ASR, also called speech-to-text / STT) maps a raw audio waveform to a text transcript.

Input. Audio is a 1-D array of amplitude samples. Almost every ASR model expects 16 kHz mono audio; resample anything else first. Models do not consume the waveform directly - it is converted to a log-mel spectrogram (a time x frequency image of energy), which the neural encoder ingests.

Output. Plain text, optionally with:

• Timestamps - per segment or per word (needed for subtitles, editing tools)
• Language ID - which language was spoken
• Translation - some models (Whisper) can emit English text from non-English speech in one pass

Neighbouring tasks (separate models, often chained with ASR):

Task	What it does	Typical tool
Voice activity detection (VAD)	Find where speech is	Silero VAD
Speaker diarization	Who spoke when	pyannote.audio
Forced alignment	Align a known transcript to audio	Qwen3-ForcedAligner, MFA
Text-to-speech (TTS)	The inverse task	see 00_Text_to_Speech

---

2. How Modern ASR Works

Four architectural eras, each still alive in some niche:

1. HMM/GMM pipelines (Kaldi era, pre-2016). Hand-built acoustic + pronunciation + language models. Legacy - only relevant for understanding older literature.
2. Self-supervised encoders + CTC (wav2vec 2.0, 2020). Pretrain an encoder on unlabeled audio, fine-tune with a CTC head. CTC predicts a character/token per frame independently - extremely fast, slightly weaker on punctuation and context.
3. Attention encoder-decoder (Whisper, 2022). A transformer decoder attends over the full encoded audio - best robustness and formatting, but slow (autoregressive) and offline-only by design (fixed 30 s windows).
4. Transducers and LLM decoders (2024-2026).
   • RNN-T / TDT (Parakeet): streams natively, near-attention accuracy at 10-50x the speed.
   • SALM / LLM-decoder (Canary-Qwen, Granite Speech, Qwen3-ASR): bolt a speech encoder onto a pretrained LLM. Current accuracy leaders - the LLM brings world knowledge to disambiguate hard audio.

Nearly all modern encoders are Conformers (convolution + self-attention) or close variants.

Decoder trade-off cheat sheet:

Decoder	Accuracy	Speed	Streaming	Example
CTC	good	fastest	yes	wav2vec 2.0
TDT / RNN-T	very good	very fast	yes	Parakeet TDT
Attention	very good	slow	no	Whisper
LLM (SALM)	best	slowest	partial	Canary-Qwen, Qwen3-ASR

---

3. Evaluation Metrics

Word Error Rate (WER) - the standard metric. Edit distance between hypothesis and reference, normalised by reference length:

\[WER = \frac{S + D + I}{N}\]

where S = substitutions, D = deletions, I = insertions, N = reference words. Lower is better; can exceed 100%.

Pitfalls:

• Normalisation dominates. Casing, punctuation and number formatting (“25” vs “twenty five”) swing WER by several points. Always compare models under the same text normalisation (the Open ASR Leaderboard uses the Whisper normaliser).
• CER (character error rate) is used for languages without whitespace word boundaries (Chinese, Japanese).
• RTFx (inverse real-time factor) = audio duration / processing time. RTFx 100 means 1 hour of audio transcribes in 36 s. The accuracy/speed trade-off is the key engineering decision.

jiwer is the standard Python package for WER:

---

import jiwer

reference = "the quick brown fox jumps over the lazy dog"
hypothesis = "the quick brown fox jumped over a lazy dog"

print("WER:", jiwer.wer(reference, hypothesis))

out = jiwer.process_words(reference, hypothesis)
print(f"substitutions={out.substitutions} deletions={out.deletions} insertions={out.insertions}")

WER: 0.2222222222222222
substitutions=2 deletions=0 insertions=0

4. The Model Landscape (mid-2026)

Whisper is no longer the accuracy leader, but remains the ecosystem default. The Open ASR Leaderboard is the reference ranking (avg WER over 8+ English test sets).

Model	Params	License	Languages	Decoder	Best for
NVIDIA Canary-Qwen 2.5B	2.5B	CC-BY-4.0	en	SALM	max English accuracy (~5.6% avg WER)
IBM Granite Speech	2-8B	Apache 2.0	en + few	LLM	accuracy per parameter (~5.3% WER)
NVIDIA Parakeet TDT	0.6-1.1B	CC-BY-4.0	en (v2), 25 (v3)	TDT	bulk transcription (RTFx ~2000)
Qwen3-ASR	0.6B / 1.7B	Apache 2.0	52 (incl. dialects)	LLM	multilingual open SOTA, language ID
Voxtral Transcribe 2	4B	Apache 2.0	13	streaming	native real-time streaming
Whisper large-v3	1.5B	MIT	99+	attention	long-tail languages, ecosystem
Moonshine	27-60M	MIT	en	attention	edge / embedded / Raspberry Pi

License nuance. MIT/Apache models are open source in the full sense. NVIDIA’s CC-BY-4.0 is a content license - commercial use is fine but attribution is required, and purists do not call it open source. Watch out: older Canary checkpoints (pre-Qwen) were CC-BY-NC (non-commercial only). Qwen3-ASR-Flash (the 52-language flagship numbers) is API-only on Alibaba Cloud; the open 0.6B/1.7B checkpoints cover fewer languages. Training data is closed for all of them - you get weights, not reproducibility.

---

5. Setup

Everything below runs on a single modest GPU (tested on a 6 GB GTX 1660 SUPER) or CPU. Package roles:

• transformers + torch - Whisper, Moonshine, Qwen3-ASR (via qwen-asr)
• datasets + torchcodec - test clips with reference transcripts (LibriSpeech)
• jiwer - WER computation
• faster-whisper - CTranslate2 Whisper runtime (optional, section 7)
• qwen-asr - Qwen3-ASR package (optional, section 8)
• nemo_toolkit[asr] - NVIDIA Parakeet/Canary (optional, heavy, section 9)

All downloads (sample clips, HF dataset cache) land in DL_tasks/datasets/, which is gitignored.

---

# Core dependencies (uncomment to install)
# %pip install -q torch transformers datasets torchcodec jiwer soundfile

# Optional per-section extras
# %pip install -q faster-whisper          # section 7
# %pip install -q qwen-asr                # section 8
# %pip install -q "nemo_toolkit[asr]"     # section 9 (heavy)
# %pip install -q sounddevice             # section 12

import time
import urllib.request
from pathlib import Path

import torch
from dotenv import find_dotenv, load_dotenv

# Knowledge/.env sets HF_TOKEN - authenticated HF Hub requests get higher rate limits
load_dotenv(find_dotenv(usecwd=True))

device = "cuda:0" if torch.cuda.is_available() else "cpu"
if device != "cpu":
    print(torch.cuda.get_device_name(0))
print("device:", device)

# All dataset downloads go to DL_tasks/datasets/ (gitignored)
DATA_DIR = Path("../../datasets")
DATA_DIR.mkdir(exist_ok=True)

# A ~14 s English sample clip (from the Qwen3-ASR repo)
SAMPLE = DATA_DIR / "sample_en.wav"
if not SAMPLE.exists():
    urllib.request.urlretrieve(
        "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav",
        SAMPLE,
    )

from datasets import load_dataset

# 73 short LibriSpeech clips with reference transcripts - our eval set
ds = load_dataset(
    "hf-internal-testing/librispeech_asr_dummy",
    "clean",
    split="validation",
    cache_dir=str(DATA_DIR / "hf_cache"),
)

def to_audio_input(audio):
    "Normalize a datasets audio entry to the dict transformers pipelines expect."
    if isinstance(audio, dict):  # datasets < 4.x
        return {"array": audio["array"], "sampling_rate": audio["sampling_rate"]}
    samples = audio.get_all_samples()  # datasets >= 4.x returns a torchcodec AudioDecoder
    return {"array": samples.data.numpy().squeeze(), "sampling_rate": samples.sample_rate}

row = ds[0]
print(to_audio_input(row["audio"])["array"].shape)
print("ref:", row["text"][:70])

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

(93680,)
ref: MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO

6. Whisper with transformers

The automatic-speech-recognition pipeline handles decoding (via ffmpeg), resampling, batching and chunking. Model sizes: tiny (39M) -> base -> small (244M) -> medium -> large-v3 (1.5B). small is a good accuracy/VRAM compromise on 6 GB cards.

---

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="openai/whisper-small", device=device)

t0 = time.perf_counter()
result = asr(str(SAMPLE), return_timestamps=True)
print(f"{time.perf_counter() - t0:.1f}s")
print(result["text"])
for chunk in result["chunks"]:
    print(chunk["timestamp"], chunk["text"])

# Long-form audio: chunk the input and batch the chunks
long_form = pipeline(
    "automatic-speech-recognition",
    model="openai/whisper-small",
    chunk_length_s=30,
    batch_size=8,
    device=device,
)

# Word-level timestamps and forced language / task selection:
result = asr(
    str(SAMPLE),
    return_timestamps="word",
    generate_kwargs={"language": "en", "task": "transcribe"},  # task="translate" -> English output from any language
)
print(result["chunks"][:5])

7. faster-whisper (CTranslate2)

The same Whisper weights on the CTranslate2 inference engine: ~4x faster than the reference implementation, with int8 quantisation that halves memory at negligible accuracy cost. The usual production choice for batch Whisper workloads.

---

# %pip install -q faster-whisper
from faster_whisper import WhisperModel

# compute_type: "float16" (GPU), "int8" (CPU or low VRAM)
fw = WhisperModel("small", device="cuda" if device != "cpu" else "cpu", compute_type="int8")

t0 = time.perf_counter()
segments, info = fw.transcribe(str(SAMPLE), beam_size=5)
print("language:", info.language, f"(p={info.language_probability:.2f})")
for seg in segments:
    print(f"[{seg.start:6.2f} -> {seg.end:6.2f}] {seg.text}")
print(f"{time.perf_counter() - t0:.1f}s")

8. Qwen3-ASR

The strongest open multilingual family (Apache 2.0, Jan 2026). 52 languages/dialects with built-in language ID, singing-voice robustness, and an optional forced aligner for timestamps. The 0.6B checkpoint fits comfortably in 6 GB; the qwen-asr package wraps transformers (and optionally vLLM for streaming/batch serving).

---

# %pip install -q qwen-asr
from qwen_asr import Qwen3ASRModel

# Note: bfloat16 needs an Ampere+ GPU; use float16 on GTX 16xx / older cards
qwen = Qwen3ASRModel.from_pretrained(
    "Qwen/Qwen3-ASR-0.6B",
    dtype=torch.float16 if device != "cpu" else torch.float32,
    device_map=device,
    max_inference_batch_size=8,
    max_new_tokens=256,
)

results = qwen.transcribe(audio=str(SAMPLE), language=None)  # language=None -> auto-detect
print(results[0].language)
print(results[0].text)

# For word/segment timestamps, load with:
#   forced_aligner="Qwen/Qwen3-ForcedAligner-0.6B", and pass return_time_stamps=True

9. NVIDIA Parakeet and Canary (NeMo)

The Open ASR Leaderboard toppers. Parakeet TDT is the speed king (RTFx ~2000 - an hour of audio in under 2 s on a datacenter GPU); Canary-Qwen 2.5B is the English accuracy king. Both CC-BY-4.0, both need the (heavy) NeMo toolkit rather than plain transformers.

---

# %pip install -q "nemo_toolkit[asr]"   # large install; needs a fairly recent GPU stack
import nemo.collections.asr as nemo_asr

parakeet = nemo_asr.models.ASRModel.from_pretrained("nvidia/parakeet-tdt-0.6b-v2")

out = parakeet.transcribe([str(SAMPLE)], timestamps=True)
print(out[0].text)
for stamp in out[0].timestamp["segment"]:
    print(f"[{stamp['start']:6.2f} -> {stamp['end']:6.2f}] {stamp['segment']}")

10. Moonshine (edge devices)

Purpose-built for embedded real-time ASR: tiny is 27M params (~27 MB quantised), and unlike Whisper it processes variable-length windows instead of always padding to 30 s - so short commands transcribe with proportionally less compute. English-only, MIT license, runs happily on CPU / Raspberry Pi.

---

moonshine = pipeline(
    "automatic-speech-recognition",
    model="UsefulSensors/moonshine-tiny",
    device=device,  # or device="cpu" - it is fast enough
)

t0 = time.perf_counter()
print(moonshine(str(SAMPLE))["text"])
print(f"{time.perf_counter() - t0:.1f}s")

11. Head-to-head Benchmark

Same clips, same normalisation, WER + wall clock per model. Caveats: this is a tiny English-only sample of clean read speech - treat it as a smoke test, not a leaderboard. For real evaluation use the full LibriSpeech test sets, or noisy/conversational sets like AMI and Earnings-22 (both on the Open ASR Leaderboard).

Measured here on a GTX 1660 SUPER (5 clips): whisper-tiny 10.6% WER / 2.4 s, moonshine-tiny 12.6% WER / 1.3 s.

---

import jiwer

N = 20  # clips to evaluate (max 73)

normalize = jiwer.Compose([
    jiwer.ToLowerCase(),
    jiwer.RemovePunctuation(),
    jiwer.RemoveMultipleSpaces(),
    jiwer.Strip(),
])

def benchmark(name, transcribe_fn):
    "Run transcribe_fn over the eval set; return (wer, seconds)."
    refs, hyps = [], []
    t0 = time.perf_counter()
    for row in ds.select(range(N)):
        hyps.append(transcribe_fn(to_audio_input(row["audio"])))
        refs.append(row["text"])
    elapsed = time.perf_counter() - t0
    wer = jiwer.wer(normalize(refs), normalize(hyps))
    print(f"{name:25s} WER {wer:6.2%}   {elapsed:5.1f}s / {N} clips")
    return wer, elapsed

results = {}

for model_id in ["openai/whisper-tiny", "openai/whisper-small", "UsefulSensors/moonshine-tiny"]:
    asr_pipe = pipeline("automatic-speech-recognition", model=model_id, device=device)
    results[model_id] = benchmark(model_id, lambda a: asr_pipe(a)["text"])
    del asr_pipe
    if device != "cpu":
        torch.cuda.empty_cache()

# faster-whisper takes (array, sr) via a file or ndarray; qwen accepts (ndarray, sr) tuples:
# results["faster-whisper-small"] = benchmark(
#     "faster-whisper-small",
#     lambda a: " ".join(s.text for s in fw.transcribe(a["array"])[0]),
# )
# results["Qwen3-ASR-0.6B"] = benchmark(
#     "Qwen3-ASR-0.6B",
#     lambda a: qwen.transcribe(audio=(a["array"], a["sampling_rate"]))[0].text,
# )

import pandas as pd

df = pd.DataFrame(
    [(k, wer, sec) for k, (wer, sec) in results.items()],
    columns=["model", "wer", "seconds"],
).sort_values("wer")
df

12. Real-time Microphone Input

Quick capture-then-transcribe loop with sounddevice. For true streaming (partial results while speaking) use a natively streaming model - Voxtral Transcribe 2, Qwen3-ASR via its vLLM backend, or Parakeet/RNN-T - or a VAD-chunked wrapper project like RealtimeSTT / whisper-streaming.

---

# %pip install -q sounddevice
import sounddevice as sd

SECONDS, SR = 5, 16000
print("recording...")
recording = sd.rec(int(SECONDS * SR), samplerate=SR, channels=1, dtype="float32")
sd.wait()
print(asr({"array": recording.squeeze(), "sampling_rate": SR})["text"])

13. Going Further

• Fine-tuning. Whisper fine-tunes well on ~5-10 h of domain audio: HF fine-tuning guide. Common Voice, FLEURS and People’s Speech are the usual open training sets.
• Domain hints without fine-tuning. faster-whisper’s initial_prompt / hotwords bias decoding toward domain vocabulary (names, jargon) for free.
• Diarization + ASR. WhisperX and pyannote.audio give “who said what” with word-level alignment.
• Forced alignment. Qwen3-ForcedAligner-0.6B (11 languages, up to 5 min) or the classic Montreal Forced Aligner.
• VAD preprocessing. Silero VAD to strip silence - big speed and accuracy win on real-world recordings.

References

• Open ASR Leaderboard and the leaderboard paper
• Whisper paper (Radford et al., 2022)
• Qwen3-ASR on GitHub
• NVIDIA Canary-Qwen 2.5B model card
• Moonshine paper
• Northflank: best open source STT in 2026 (benchmarks)

---