Project

Speech Recognition on Noisy Radio Messages

An evaluation of whether Whisper speech recognition keeps safety-critical facts correct in simulated noisy radio distress calls.

View repository

Overview

An average word error rate (WER) can hide errors in the few words that matter. In a distress call these are the vessel name, the position, the nature of distress and the number of persons on board. A missing fact is noticed and asked again, but a wrong fact looks plausible and is dangerous. This project tests whether WER shows if these facts survive speech recognition on a simulated noisy radio channel.

What I built

  • Generated 30 synthetic MAYDAY-style messages from one template with a fixed seed, so every fact is known.
  • Turned the messages into speech with Piper TTS and three public-domain voices.
  • Simulated a radio channel with a 300 to 3000 Hz band-pass filter and white noise from 20 to 0 dB SNR.
  • Transcribed the audio with faster-whisper tiny.en and base.en using deterministic decoding.
  • Extracted the four facts with rule-based patterns and scored each as correct, missing or wrong.
  • Reported WER, CER and fact error rates with 95% bootstrap confidence intervals.
  • Checked hallucinations on noise-only clips.
  • Added tests, CI, a FastAPI endpoint and a Docker image.

Key results

  • With base.en, WER rose tenfold from clean speech (0.12) to a 0 dB channel (1.22).
  • Over the same range the wrong-fact rate stayed between 12.5% and 20.0%, so WER did not track wrong facts.
  • Noise mostly turned facts into missing ones: 5.0% missing on clean speech and 73.3% at 0 dB.
  • Vessel names and positions carried most wrong facts; the distress type was never wrong.
  • base.en produced text on 7 of 30 noise-only clips; tiny.en on none.

Limitations

  • Synthetic speech and a simulated channel, not real VHF radio.
  • One fixed template, English only and 30 messages, so confidence intervals are wide.
  • Strict matching counts spelling variants of names as wrong.

Tools

  • Python
  • faster-whisper
  • Piper TTS
  • SciPy
  • FastAPI
  • pytest
  • GitHub Actions
  • Docker
View repository