Figure 1. Headline result on real on-air recordings with real band noise added (error bars: 95% bootstrap CI). Both new models beat ft6 and DeepCW by 8.4–10.3 percentage points (paired bootstrap, p < 0.001).
Part 1 introduced NeuroCW, my end-to-end neural CW decoder, and its previous best model ft6, trained only on synthetic CW. Part 2 asked for real recordings, because no real-air number existed. This report closes that gap: a real-recording benchmark, a new model generation (CW3), and a final fine-tune on real KiwiSDR audio whose text is known from published sources. Two models came out of it:
- NeuroCW CW3-L-FT (55.6M parameters): the large model, for PCs with a GPU.
- NeuroCW CW3-Lite-FT (5.25M parameters): the light model, for CPU-only PCs.
Everything below is from training logs and the benchmark output. Estimates are labelled as estimates.
1. What changed since ft6
| ft6 (previous best) | CW3-L-FT / CW3-Lite-FT | |
|---|---|---|
| Input band | 156–1406 Hz | 203–2297 Hz |
| Attention | no QK-norm (attention collapse found) | QK-norm on q, k before RoPE |
| Learning rate (large) | – | 2e-4 (cw3-large used 3e-4) |
| Training data | synthetic only | synthetic pre-train + real on-air fine-tune |
| Real-air CER | never measured until now | measured on 6 real sets |
| CPU model | – | 5.25M Lite model, 126× real time |
Table 1. ft6 versus the new CW3 models.
Figure 2. The full pipeline. The data side ran on a PC and public KiwiSDR receivers; the training side ran on one NVIDIA DGX Spark (GB10). Grey boxes are next steps and are not done yet.
2. Architecture
Figure 3. CW3 is a Conformer-CTC model: fixed STFT front-end (5 ms hop), 4× conv subsampling to 20 ms frames, a language embedding (en/ko/ja), N Conformer blocks and a 145-symbol CTC head with greedy decoding.
Why QK-norm and a lower LR? In the CW3 generation, the 55.6M large model lost to the 5.25M lite model on every set (bench_en 6.16% vs 4.20%). I traced two causes:
- Attention-entropy collapse. Every older model with d_model ≥ 256 (cw2 main/large/ft6, cw3-large) had max attention logits of 300–2,757, entropy of 0.03–0.5 nat (nearly one-hot) and attention gates closed at about 0.02. It started near step 20k, when the SNR curriculum reached low SNR and the gradient norm spiked (2.7 → 18–38). With QK-norm, the max logit stayed near 12 and entropy above 3.6 nat.
- LR too high for the width. QK-norm alone was not enough. Following the μP-style rule of scaling LR inversely with width (5e-4 × 144/384 ≈ 2e-4), the large model's train CTC loss fell to 0.622, below lite's 0.704, and it took first place on every synthetic set.
3. Data pipeline: 13 days of receiving
Ground-truth rule: a clip becomes a label only if its text can be checked against a published or deterministic source: ARRL W1AW bulletins (text published on arrl.org), the NCDXF/IARU beacon schedule, or my own synthetic generator. No decoder output, from ft6 or any other model, is ever used as a label. Decoders were used only to exclude suspicious clips.
Figure 4. The data funnel. 332.9 receiver-hours from 93 KiwiSDR channels produced 1,145 verified clips (4.1 h). The real-audio training pool is approximately 2.4 h; the val and test bulletins are held out.
| Collection item | Value |
|---|---|
| Period | Nov 1 23:30 → still running Nov 14 23:20 KST (13 days) |
| Parallel receiver workers | 8 |
| KiwiSDR channels / connections | 93 / 1,157 (46 receivers delivered W1AW bulletins) |
| Audio recorded | 19,971 receiver-minutes ≈ 332.9 h |
| W1AW bulletins | 137.8 h |
| W1AW code practice | 96.1 h (not used: practice texts not available) |
| Real band-noise bank | 62.6 h, 13,353 chunks |
| NCDXF beacons / DK0WCY | 13.9 h / 22.5 h (DK0WCY not verifiable, 0 written) |
| Early decoder-labelled clips (Nov 1–2) | 1,136, never used (not ground truth) |
Table 2. Collection numbers.
Alignment. The published bulletin text is rendered into the exact 18-WPM keying pattern and slid against the measured tone envelope at 4 ms resolution. Each clip gets the text whose pattern it matches.
Figure 5. The v3 strict gate. A clip must pass all four checks; the counts are round-1 W1AW rejections (one clip can fail several). An independent DSP decoder, not a neural model, provides check 2.
| Folder | Exported | After round 1 | Final | Hours |
|---|---|---|---|---|
| Real W1AW | 1,912 | 1,108 | 1,108 | 4.08 |
| Real NCDXF | 90 | 37 | 37 | 0.03 |
| NCDXF carrier-only | 188 | 0 | 0 | – |
| Synthetic mix | 2,000 | 1,183 | 1,079 | 4.13 |
| Noise (CW / SSB) | 5,857 / 7,532 | 5,846 / 7,527 | 4,845 / 6,979 | 13.46 / 19.39 |
| Total | 17,579 | 15,701 | 14,048 | 41.1 |
Table 3. The Nov 14 re-filter. Round 2 removed noise clips that turned out to contain weak CW (one held a real contest exchange, TEST JF3VCH OF3KCH) and synthetic clips with tones above the 2297 Hz front-end band.
Leak-free split, by bulletin: train ARLP045 (652 W1AW clips + 7 NCDXF), val ARLP044 (311 clips), test ARLD044 (64 clean clips; ×6 real-noise variants at 0/−6/−12 dB = 384). No sentence, multi-receiver group or noise clip appears in more than one split, and the receiver that supplied most of the training audio is kept on the train side only. On the final real training pool, the best decoder matched 648 of 652 W1AW clips exactly; each of the other 4 differs by one character at a weak spot.
4. Training on the DGX Spark
All training ran on one NVIDIA DGX Spark (GB10 GPU), one run at a time. Each base model trained for 150k steps on synthetic CW generated on the fly (en/ko/ja text, real band-noise mixing, SNR curriculum). Each fine-tune ran 5k steps on a mixed batch of real and synthetic audio.
Figure 6. Pre-training. Left: synthetic validation score for every CW3-generation run. The dotted line is the failed cw3-large continuation, which got worse after its LR was re-warmed. Right: logged GPU-hours per run.
| Hyper-parameter | Pre-training (base) | Real-data fine-tune |
|---|---|---|
| Steps | 150,000 | 5,000 |
| Batch | 480 s of audio | 480 s of audio |
| Optimizer | AdamW, wd 0.01, clip 1.0 | same |
| LR (large / lite) | 2e-4 / 5e-4 | 5e-5 / 2e-4 |
| Schedule | warmup 6,000, cosine to 2% | warmup 300, cosine to 5% |
| EMA | 0.9995 | 0.999 |
| Batch mix | synthetic + SNR curriculum | 30% real W1AW/NCDXF, 10% collector synth, 8% pure noise (empty target), rest synthetic |
| Checkpoint selection | synthetic val | real val (ARLP044), guard: synthetic may not regress > 3% |
Table 4. Training recipe, taken from the run configs. Pure-noise clips with an empty target teach the model not to invent characters when no signal is present. The key-state loss is switched off for real clips, which have no exact key timing.
Figure 7. The fine-tune sweep (7 runs, 5,000 steps each; 6 shown). Left: real-noise validation CER falls steadily. Right: the synthetic score stays roughly flat. Distillation (grey dotted) was rejected.
| Final model | Base run | Base GPU time | Fine-tune | Total |
|---|---|---|---|---|
| CW3-L-FT | large + QK-norm, LR 2e-4 | 35.6 h | 1.2 h | ≈36.8 h |
| CW3-Lite-FT | lite + QK-norm | 18.6 h | 0.9 h | ≈19.5 h |
Table 5. DGX Spark GPU time for the two final models.
A note on checkpoint choice: the 3% guard rejected CW3-L-FT's last step (synthetic ja 8.25 → 8.90). I kept the last step for the GPU model because it was best on real noisy val and test. A conservative alternative, CW3-L-FT-b, gives up no synthetic accuracy and was benchmarked too (see the full benchmark heatmap below).
5. Benchmark: 10 models × 14 sets
All models read the same WAV files and were scored by the same code. CER is the Levenshtein distance divided by the number of reference symbols; spaces count, and a prosign counts as one symbol. Confidence intervals come from a 2,000-sample clip bootstrap. Model differences use a 5,000-sample paired bootstrap. DeepCW ran its official ONNX model on CPU, resampled to 3200 Hz as in its official example.
Figure 8. Real recordings, the fair comparison. All tones are near 650 Hz, inside every model's input band. Right: characters invented on 150 carrier-only NCDXF clips, where no text was sent.
| Set | DeepCW | ft6 | cw3-large | CW3-L-FT | CW3-Lite-FT |
|---|---|---|---|---|---|
| W1AW + real noise (test, 384) | 28.82 | 29.14 | 25.94 | 18.83 | 20.39 |
| W1AW clean (64) | 1.29 | 0.30 | 0.08 | 0.00 | 0.00 |
| NCDXF v3 (37) | 1.89 | 3.14 | 2.52 | 0.63 | 1.26 |
| NCDXF legacy (73) | 1.55 | 5.88 | 5.26 | 2.48 | 1.86 |
| False chars, carrier-only (150) | 228 | 93 | 112 | 67 | 32 |
| Synthetic bench_en (450) | 19.17 | 5.74 | 6.16 | 3.35 | 4.43 |
| Params / CPU speed | ≈3.8M / 153× | 55.3M / 35× | 55.6M / 36× | 55.6M / 39× | 5.25M / 126× |
Table 6. Key results, CER % (false characters as a count). Paired bootstrap on the real-noise test, all p < 0.001: CW3-L-FT −9.98 pp vs DeepCW (95% CI [−11.44, −8.57]) and −10.31 pp vs ft6; CW3-Lite-FT −8.43 pp vs DeepCW and −8.76 pp vs ft6. On NCDXF legacy, DeepCW is lowest, but the gap is not significant (CW3-L-FT − DeepCW = +0.93 pp, CI [−0.63, +2.81]).
Figure 9. Real W1AW bulletins mixed with real band noise recorded at a different time on a different receiver. The gap opens as noise rises: at −6 dB, CW3-L-FT scores 9.3% against 16.0% for DeepCW and 26.1% for ft6.
Figure 10. The full benchmark: 10 models (including the ablation checkpoints) × 14 sets, 2,665 clips, 9.0 h of audio.
Figure 11. The DeepCW-protocol synthetic benchmark by SNR. At −10 dB, DeepCW scores 20.9% against 1.7% for CW3-L-FT, roughly a 4 dB SNR advantage at equal error rate. This set favors the CW3 family (see caveats).
Figure 12. The same synthetic benchmark by keying speed, 12–40 WPM.
Figure 13. Size against CPU speed (20 s clip, 2 threads, fp32, the ARM CPU of the GB10 system). CW3-Lite-FT is close to DeepCW in size and speed but has much lower real-noise CER. int8 dynamic quantization gave no speed gain on the light model, so I did not use it.
Ablation: what each step contributed
Figure 14. The ablation. QK-norm plus the lower LR fixed the large model on synthetic data; the real-data fine-tune is what moved the real-noise CER. Fine-tune effect: large −4.07 pp, lite −3.61 pp (p < 0.001), with no significant change on bench_en.
Korean and Japanese synthetic sets got about 0.3 pp worse for the large model and 0.4–0.5 pp worse for the light model, within the confidence intervals. That is the cost of fine-tuning on English-only real data.
- The synthetic sets favor the CW3 family. They come from the same generator as the CW3 training data: in-distribution for CW3, out-of-distribution for DeepCW, and partly out-of-distribution for ft6.
bench_enreproduces DeepCW's SNR convention; it is not DeepCW's official data. The fair comparison is on real recordings. - The real-noise set somewhat favors the CW3 family. The added noise comes from the same collection campaign as the training noise, with no clip overlap. DeepCW and ft6 never saw this noise.
- The clean W1AW and NCDXF sets have no such bias, and there the gaps are small. On NCDXF, DeepCW is statistically on par.
- Small samples. The test bulletin ARLD044 has only 64 clips; the noisy set is 6 noise variants of those clips, so it is not fully independent. NCDXF v3 is 37 clips / 159 characters, so one error is 0.63 pp.
- W1AW is 18-WPM machine keying. Performance on hand-keyed, fast or contest CW is estimated only from synthetic sets.
- Input bands. DeepCW sees only 400–1200 Hz; ft6 works only at ≤ 1400 Hz (front-end 156–1406 Hz); CW3 covers 203–2297 Hz.
ssb_hi_en(1500–2300 Hz) lies entirely outside the DeepCW and ft6 bands, and parts ofsmixdo too, so their high CER there reflects band limits, not decoding ability. The main tables score every set in full; Appendix B of the technical report re-scores only tones inside 400–1200 Hz for all models, and the ranking is unchanged. - DeepCW was fed whole clips of up to about 20 s. I did not check whether that window length is optimal for DeepCW.
6. Real decode samples
Real test clips with the reference text (shaded) and each model's raw output. Wrong or extra characters are shown in red, and · marks a missing character. The percentage is the per-clip CER. I included clips where the new models lose, too.
| D44PX FROM BOA VISTA (clean reception, 13.5 s) | |
| DeepCW | D44PX FROM EBOA VISTA (5.0%) |
| ft6 | D44PX FROM BOA VISTA (0.0%) |
| CW3-L-FT | D44PX FROM BOA VISTA (0.0%) |
| CW3-Lite-FT | D44PX FROM BOA VISTA (0.0%) |
| AND WA7BNM WEB SITES. (real noise added at −6 dB, 12.6 s) | |
| DeepCW | AND WA7BNM WEB SITES. (0.0%) |
| ft6 | AND WA7BNM WEB SITES. (0.0%) |
| CW3-L-FT | AND WA7BNM WEB SIVES. (4.8%) |
| CW3-Lite-FT | AND WA7BNM WEB SITES. (0.0%) |
| NOVEMBER 6 AND 7, (real noise added at −6 dB, 11.2 s) | |
| DeepCW | N7VTB·R 6 A1 7, (35.3%) |
| ft6 | N7UABC OK6M· ·DS, (76.5%) |
| CW3-L-FT | NOUESTBER O AND S, (29.4%) |
| CW3-Lite-FT | NOUSTBER D AND 7S, (29.4%) |
| D44PX FROM BOA VISTA (real noise added at −12 dB, 13.5 s) | |
| DeepCW | D44PX FROM BOA VBSN (15.0%) |
| ft6 | D44PX FROU BOA VIS4A (10.0%) |
| CW3-L-FT | D44PX FROU BOA VISTA (5.0%) |
| CW3-Lite-FT | D44PX FROU BOR VISTA (10.0%) |
| USING VARIOUS MODES. (real noise added at −12 dB, 12.7 s) | |
| DeepCW | QKW7K (100.0%) |
| ft6 | (no output) (100.0%) |
| CW3-L-FT | ·N· ·OM ·OON (75.0%) |
| CW3-Lite-FT | (no output) (100.0%) |
| ZL6B (NCDXF beacon, 3.1 s) | |
| DeepCW | ZL6B (0.0%) |
| ft6 | ZL6B (0.0%) |
| CW3-L-FT | ZL6B (0.0%) |
| CW3-Lite-FT | ZL6B (0.0%) |
Table 7. Raw greedy-CTC outputs on real ARLD044 / NCDXF clips (from the benchmark's per-clip output files). At −12 dB, every model fails on some clips.
7. The effort: every hour on the record
Figure 15. The whole project on one time axis, Oct 6 → Nov 14, 2026 (KST). Top: every GPU run on the DGX Spark. Middle: the data side. Bottom: cumulative GPU-hours.
Figure 16. Where the GPU time went. More than half of it went into generations or experiments that did not become the final model.
| Effort item | Value |
|---|---|
| Project span on the training server | Oct 6 → Nov 14, 2026 (≈39.6 days) |
| GPU training, logged (26 runs) | 303.2 GPU-hours (≈12.6 days of non-stop GPU time) |
| + first-generation runs, Oct 6–9 | ≈52.9 h (estimate from checkpoint file times; a lower bound) |
| Total GPU training, all generations | ≈356 GPU-hours (≈14.8 days) |
| Optimizer steps actually run (logged runs) | 1,522,832 |
| cw2 generation (incl. ft6) / cw3-large + failed continuation / CW3 pre-training / fine-tuning | 136.70 / 50.74 / 107.66 / 8.13 GPU-h |
| Real-signal collection | 13 days non-stop, 8 parallel receivers, 93 channels, 332.9 receiver-hours |
| Clips handled in curation | 17,579 exported → 14,048 kept; real labelled 2,190 → 1,145 |
| Collector incidents fixed | dead worker + memory errors (Nov 8), alignment bug (Nov 12), receiver-list outage (Nov 14) |
| Final benchmark | 10 models × 14 sets = 2,665 clips, 9.0 h of audio |
Table 8. Effort totals. Not included: my own planning and debugging time, CPU time for on-the-fly data generation (part of each run's wall-clock), and the scoring passes after each run (about 0.5 h each in cw2).
What failed (and what it cost)
- Re-warming cw3-large from 150k steps (planned to 300k): val went from 0.116 to 0.133 by step 218k. I stopped it after 15.8 GPU-hours. A collapsed model cannot be fixed by more training.
- An alignment bug hid already-collected data for days. On Nov 12, only 40 W1AW clips were in train, against 1,153 in review plus 90 "unmatched" recordings. After the fix, re-aligning the 90 recordings recovered 1,416 clips.
- A loose first gate (align ≥ 0.90): about half the real clips did not survive the strict audit.
- "Noise" clips with weak CW inside: 1,549 removed after three different decoders agreed there was a signal. Agreement was used to exclude clips, never as a label.
- Synthetic tones above the front-end band (> 2297 Hz) cannot be learned and poison training, so the generator is now capped at 2300 Hz.
- Distillation, lite ← large: the synthetic score got 10% worse. Rejected.
8. Limitations
- The real test set is one bulletin (64 clips) of 18-WPM machine keying. That is a narrow slice of real CW.
- The fine-tune data is English only. Korean and Japanese real-air accuracy is still unmeasured.
- The light model nearly fails when two signals overlap (for example, 703 Hz and 781 Hz). A user-selected band-pass of about ±60 Hz in front of it brings it back to near the large model.
- Everything here was decoded offline in whole clips. Streaming behavior has not been tested yet.
9. What's next: test → ONNX → app
- Test. Score both models on held-out ARLD (DX) bulletins, now being collected Wednesday to Friday into a never-train folder, and add more bulletins over time for text diversity.
- ONNX. Export CW3-Lite-FT (and CW3-L-FT), and check numerical parity with PyTorch on all 14 sets and CPU speed.
- App. Integrate into the NeuroCW application with overlapping 16–20 s windows, streaming at 2–4 s latency, and a tone-select band-pass for the light model.
Until those steps are done, there is no new NeuroCW release. This post reports training results only.
10. Open data, and a call for real CW recordings
11. Credits, licenses and references
- DeepCW, e04: github.com/e04/web-deep-cw-decoder (model distributed as deepcw-engine, AGPL-3.0). The official
model.onnx/model.onnx.jsonwere used unmodified, run with onnxruntime on CPU, for comparison only. - Ground-truth text: ARRL W1AW bulletins (arrl.org/w1aw) and the NCDXF/IARU International Beacon Project schedule (ncdxf.org/beacon).
- Audio: public KiwiSDR receivers (kiwisdr.com). Thanks to every operator who keeps a receiver open to the world.
- A. Graves et al., "Connectionist Temporal Classification," ICML 2006. PDF
- A. Gulati et al., "Conformer: Convolution-augmented Transformer for Speech Recognition," Interspeech 2020. arXiv:2005.08100
- J. Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding," 2021. arXiv:2104.09864
- A. Henry et al., "Query-Key Normalization for Transformers," Findings of EMNLP 2020. arXiv:2010.04245
- G. Yang et al., "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer" (μP), 2022. arXiv:2203.03466
NeuroCW series: Part 1: Data, Model and DGX Spark Training · Part 2: Call for Real On-Air CW Recordings · On-Air Test (KiwiSDR) · Korean Test · Japanese Test
All posts: NeuroCW · NeuroSDR
73, Ian Lee / KD8CEC
Comments
Post a Comment