Preliminary Test: Small Local Speech-to-Text on Real HF Radio (Shortwave Broadcast and Ham SSB)

Speech-to-text models are usually measured on clean podcasts and audiobooks. HF radio is different: fading, QRM, narrow SSB passbands, net-style exchanges and low SNR. I wanted a plain answer to one question — how do small, GPU-friendly ASR models handle real KiwiSDR audio?

Preliminary test, not a final benchmark. This is a first validation pass of speech-to-text for shortwave broadcast and amateur radio use. The reference transcripts were built by cross-checking each broadcast with two Whisper models, so they may not be fully accurate. For a more rigorous check, I will record shortwave broadcasts whose content (script or transcript) is published, and run a second comparison against that known text.

Dataset

Kind	Clips	Notes
English shortwave broadcast (SWBC)	14	WRMI / WINB / WWCR English segments, KiwiSDR captures
English ham SSB	11	VK40 net ~7065 kHz (Australian KiwiSDR), audible speech only
Total scored	25	Pure-noise / hallucination-only clips dropped

Audio was resampled to 16 kHz mono with soxr. No denoising was applied.

Methodology — read this first

References were built from two Whisper drafts (large-v3-turbo and large-v3): pure-noise hallucinations such as “Thank you.” and loops were rejected, the draft with the higher shared content was preferred, and only light cleanup was applied. No single model output was treated as ground truth.

Caveat: because the references came from Whisper, Whisper’s own scores are not independent. The 0.000 SW WER for Whisper large-v3 is an artifact of reference construction, not a real win. The fair comparison is among the three non-Whisper engines.

Preliminary ranking: local ~0.6–1B engines (word-weighted)

Engine	All WER	All CER	SW WER	SW CER	SSB WER	SSB CER
NVIDIA Parakeet-TDT-0.6b-v2	0.229	0.163	0.072	0.046	0.443	0.322
NVIDIA Canary-1b-v2	0.282	0.220	0.089	0.058	0.545	0.440
FunASR SenseVoice-Small	0.335	0.260	0.152	0.092	0.585	0.487

Parakeet > Canary > SenseVoice. Shortwave broadcast is easy for all three; ham SSB is where they all struggle.

WER/CER, all 25 clips (Whisper bars are reference-contaminated)

Shortwave broadcast, 14 clips

Ham SSB, 11 clips

Per-clip WER heatmap

Whisper rows (contaminated — reference only)

Engine	All WER	All CER	SW WER	SW CER	SSB WER	SSB CER
Whisper large-v3-turbo	0.086	0.056	0.049	0.034	0.135	0.086
Whisper large-v3	0.088	0.065	0.000	0.000	0.207	0.154

Running it safely

Local models ran on a DGX Spark-class GPU box with a hard abort if available RAM fell below 40 GB (never tripped; minimum ~114.9 GB available). Peak CUDA allocation was ~4.8 GB (Parakeet); Canary used ~1.8–1.9 GB. Parakeet needed one process per clip to avoid a sticky CUDA illegal-address error. The 30B Nemotron transcribe model was deliberately not loaded.

Limitations

• SSB clips come mostly from one VK40 net on ~7065 kHz.
• Whisper-derived references bias Whisper scores.
• References come from two-Whisper cross-checking, not a known script; they may contain errors. A second round with content-published broadcasts is planned.
• No denoise arm; no Nemotron-3.5-Transcribe-30B.
• English only; non-English SW was excluded.

Open data

The metrics CSV and figures will be released alongside this post so anyone can check the numbers. Next step: a second comparison using shortwave broadcasts with published content as true references, then human-verified SSB references and a denoise comparison.

73,
Ian Lee, KD8CEC