In my first preliminary test, the answer key came from two Whisper models, so it could be wrong in places. This time the key comes from broadcasts whose words are known in advance: the WWV and WWVH time stations. Their announcements follow a published format, and the space-weather bulletin text comes from NOAA. That gives a fair answer key without anyone having to guess.
Method
- 42 WWV/WWVH AM recordings captured through public web SDRs.
- Expected-content key for each clip, built from the published broadcast format and NOAA bulletins.
- Only 8 clips have clear enough voice to pass the quality filter. They are the headline set.
- WER with 95% clip-bootstrap confidence intervals (10,000 resamples).
Result 1: accuracy on clean clips (preliminary)
| Engine | WER | 95% CI |
|---|---|---|
| Nemotron-3.5 transcribe 30B (FP8) | 0.231 | 0.114 to 0.371 |
| Parakeet TDT 0.6B v2 | 0.310 | 0.195 to 0.437 |
| Whisper large-v3 turbo | 0.336 | 0.217 to 0.464 |
| Nemotron-3.5 ASR streaming 0.6B | 0.392 | 0.238 to 0.557 |
| Whisper large-v3 | 0.455 | 0.207 to 0.762 |
| SenseVoice small | 0.571 | 0.447 to 0.700 |
| Canary 1B v2 | 0.730 | 0.622 to 0.839 |
With only 8 clips, the top five overlap. Read this as a direction, not a verdict.
Result 2: hallucination on weak AM
Across all 42 clips, including the weak and noisy ones, the broadcasts contained 2,946 words. Nemotron-30B produced 9,716. When the signal faded, it went into loops until it hit the token limit, invented whole stories (a courtroom trial, a meeting roll call), or described the audio instead of transcribing it.
Whisper with fair decoding settings hallucinated less. WER on all 42 clips fell from 1.02 to 0.87 for v3 and from 1.34 to 0.91 for turbo. The familiar "Thank you." loops were still there.
Lesson for radio use: on a weak signal, a fluent-looking transcript is not proof that anything was heard.
Result 3: an AI-made answer key is not trustworthy
Using the strongest model's output as a "silver" answer key looks convenient. Scored against the gold key, that silver key had a WER of 3.19 (95% CI 1.51 to 5.32). Engine rankings under silver and gold agreed only at Spearman rho 0.64 (p = 0.086, 8 engines).
What comes next
- More WWV/WWVH gold clips until November 9, followed by an updated table.
- SSB (amateur radio voice) is outside this study and will be covered separately once a proper gold key exists.
73,
Ian Lee, KD8CEC
Comments
Post a Comment