Project
Speech Recognition on Noisy Radio Messages
An evaluation of whether Whisper speech recognition keeps safety-critical facts correct in simulated noisy radio distress calls.
View repositoryOverview
An average word error rate (WER) can hide errors in the few words that matter. In a distress call these are the vessel name, the position, the nature of distress and the number of persons on board. A missing fact is noticed and asked again, but a wrong fact looks plausible and is dangerous. This project tests whether WER shows if these facts survive speech recognition on a simulated noisy radio channel.
What I built
- Generated 30 synthetic MAYDAY-style messages from one template with a fixed seed, so every fact is known.
- Turned the messages into speech with Piper TTS and three public-domain voices.
- Simulated a radio channel with a 300 to 3000 Hz band-pass filter and white noise from 20 to 0 dB SNR.
- Transcribed the audio with faster-whisper
tiny.enandbase.enusing deterministic decoding. - Extracted the four facts with rule-based patterns and scored each as correct, missing or wrong.
- Reported WER, CER and fact error rates with 95% bootstrap confidence intervals.
- Checked hallucinations on noise-only clips.
- Added tests, CI, a FastAPI endpoint and a Docker image.
Key results
- With
base.en, WER rose tenfold from clean speech (0.12) to a 0 dB channel (1.22). - Over the same range the wrong-fact rate stayed between 12.5% and 20.0%, so WER did not track wrong facts.
- Noise mostly turned facts into missing ones: 5.0% missing on clean speech and 73.3% at 0 dB.
- Vessel names and positions carried most wrong facts; the distress type was never wrong.
base.enproduced text on 7 of 30 noise-only clips;tiny.enon none.
Limitations
- Synthetic speech and a simulated channel, not real VHF radio.
- One fixed template, English only and 30 messages, so confidence intervals are wide.
- Strict matching counts spelling variants of names as wrong.
Tools
- Python
- faster-whisper
- Piper TTS
- SciPy
- FastAPI
- pytest
- GitHub Actions
- Docker