Audio Transformers¶

In a nutshell

Audio transformers clean up and inspect the sound from your microphone before the assistant tries to turn it into words. Like a sound engineer adjusting a recording, they can do things such as reduce background noise or detect which language is being spoken, which helps the assistant understand you more reliably. See Transformer Plugins for the wider family and the Glossary for unfamiliar terms.

📐 Formal specification

Audio transformers are the audio chain of OVOS-TRANSFORM-1 — Transformer Plugins §3.1 (a formal architecture spec). The spec's pre-STT injection point takes a raw audio chunk plus an audio-format metadata object (sample rate, width, channels) and an optional lang, and returns a (possibly rewritten) chunk, updated metadata, and lang. This is the natural place to set session.detected_lang from acoustic features (§7.1). A transformer that changes the audio's physical format MUST update the metadata to match. Ordering: the chain runs by ascending priority (lowest first), matching the spec.

Audio Transformers in OpenVoiceOS (OVOS) are plugins designed to process raw audio input before it reaches the Speech-to-Text (STT) engine. They enable functions such as noise reduction, language detection, and data transmission over sound.

Processing Flow¶

The typical audio processing pipeline in OVOS is as follows:

Audio Capture: Microphone captures raw audio input.
Audio Transformation: Audio Transformers preprocess the raw audio.
Speech-to-Text (STT): Transformed audio is converted into text.
Intent Recognition: Text is analyzed to determine user intent.

Audio Transformers operate in step 2, allowing for enhancements and modifications to the audio signal before transcription.

They run inside the listener (e.g. ovos-dinkum-listener), driven by the voice loop. As audio is captured, the loop feeds chunks to each loaded transformer (feed_hotword/feed_speech populate internal buffers). Just before STT, the service calls each transformer's transform(audio_data), which returns (audio_data, context). Any returned context is merged into the recognize_loop:utterance message. Transformers run in ascending priority order (lower priority first), and reset() clears the buffers at the end of each cycle.

Configuration¶

To enable Audio Transformers, add them to your mycroft.conf under the audio_transformers section:

"audio_transformers": {
  "plugin_name": {
    // plugin-specific configuration
  }
}

Replace "plugin_name" with the identifier of the desired plugin and provide any necessary configuration parameters.

Available Audio Transformer Plugins¶

OVOS GGWave Audio Transformer¶

Purpose: Enables data transmission over sound using audio QR codes.
Features:
- Transmit data such as Wi-Fi credentials, URLs, or commands via sound.
- Integrates with the ovos-skill-ggwave for voice-controlled activation.
Installation:

  pip install ovos-audio-transformer-plugin-ggwave

Configuration Example:

  "audio_transformers": {
    "ovos-audio-transformer-plugin-ggwave": {
      "start_enabled": true
    }
  }

A listen_timeout key (seconds, default 300) auto-disables the listener after it has been enabled over the bus with ovos.ggwave.enable, so a stray or forgotten enable does not leave data-over-sound listening on forever. Set it to 0 or negative to listen indefinitely once enabled. It has no effect on start_enabled: enabling the listener via config is an explicit operator decision to run always-on, and does not arm the timer.

For more information, visit the GitHub repository.

OVOS SpeechBrain Language Detection Transformer¶

Purpose: Automatically detects the language of spoken input to route it to the appropriate STT engine.
Features:
- Subclasses AudioLanguageDetector, so its transform attaches stt_lang and lang_probability to the message context for downstream language routing.
- Uses SpeechBrain models for language identification.
- Adds multilingual support by dynamically selecting the correct language model.
Installation:

  pip install ovos-audio-transformer-plugin-speechbrain-langdetect

Configuration Example:

  "audio_transformers": {
    "ovos-audio-transformer-plugin-speechbrain-langdetect": {}
  }

For more information, visit the GitHub repository.

OVOS Band-pass Audio Transformer¶

Purpose: Attenuates energy outside a configurable pass-band on the captured speech audio before STT, removing non-phonetic noise (low-frequency rumble, high-frequency hiss) without touching the speech formants.
Features:
- Defaults to the 300–3400 Hz telephone speech band.
- Configurable Butterworth filter order and sample rate; band edges are clamped below Nyquist so an aggressive band on a low sample rate degrades gracefully instead of failing.

In practice this reduces the STT engine's sensitivity to out-of-band noise (traffic, fans, hum) that falls outside the speech band, at the cost of also discarding whatever spectral content of the voice itself sits outside that range.

Installation:

  pip install ovos-audio-transformer-plugin-bandpass

Configuration Example:

  "audio_transformers": {
    "ovos-audio-transformer-plugin-bandpass": {
      "low_hz": 300,
      "high_hz": 3400,
      "order": 4,
      "sample_rate": 16000
    }
  }

For more information, visit the GitHub repository.

Writing your own Audio Transformer¶

Subclass AudioTransformer (ovos_plugin_manager.templates.transformers) and override transform(audio_data) -> Tuple[bytes, dict], the single method the STT stage calls. Register the class under the opm.transformer.audio entry-point group. The base class also offers optional hooks (on_audio, on_hotword, on_speech, on_speech_end) that let you inspect audio earlier in the voice loop. Override only the ones you need.

This class allows you to process raw audio chunks at various stages before the Speech-to-Text (STT) engine processes the audio.

Base Class Overview¶

Your custom transformer should subclass:

from ovos_plugin_manager.templates.transformers import AudioTransformer

class MyCustomAudioTransformer(AudioTransformer):
    def __init__(self, name="my-custom-audio-transformer", priority=10, config=None):
        super().__init__(name, priority, config)

    def on_audio(self, audio_data):
        # Process non-speech audio chunks (e.g., noise)
        return audio_data

    def on_hotword(self, audio_data):
        # Process full hotword/wakeword audio chunks
        return audio_data

    def on_speech(self, audio_data):
        # Process speech audio chunks during recording (not full utterance)
        return audio_data

    def on_speech_end(self, audio_data):
        # Process full speech utterance audio chunk
        return audio_data

    def transform(self, audio_data):
        # Optionally perform final transformation before STT stage
        # Return tuple (transformed_audio_data, optional_message_context)
        return audio_data, {}

Lifecycle & Methods¶

Initialization: Override initialize() for setup steps.
Audio Feed Handlers:
- on_audio: Handle background or non-speech chunks.
- on_hotword: Handle wakeword/hotword chunks.
- on_speech: Handle speech chunks during recording.
- on_speech_end: Handle full utterance audio.
Final Transformation:
- transform: Return the final processed audio and optionally a dictionary of additional metadata/context that will be passed along with the recognize_loop:utterance message.
Reset: The reset() method clears internal audio buffers, called after STT completes.

Priority¶

Transformers run in ascending priority order (lower priority first). The default is 50. Pick a low priority for early-stage cleanup (noise reduction) that later transformers should see, and a high priority for anything that should run last.

Plugin Registration¶

A full pyproject.toml for a standalone plugin package:

[project]
name = "ovos-audio-transformer-mycustom"
version = "0.1.0"
dependencies = ["ovos-plugin-manager"]

[project.entry-points."opm.transformer.audio"]
"my-custom-audio-transformer" = "my_module:MyCustomAudioTransformer"

An opm.transformer.audio.config group is also available, for a dict of config metadata an installer or GUI can read. It is optional. Add it once the plugin has settings worth advertising. The legacy alias neon.plugin.audio is still recognized for the entry-point group by the plugin loader, but new plugins should use opm.transformer.audio (the GGWave and SpeechBrain transformers above both already do).

Configuration Example¶

Add your transformer to mycroft.conf:

"audio_transformers": {
  "my-custom-audio-transformer": {
    // plugin-specific config options here
  }
}

Test it without OVOS¶

AudioTransformer subclasses are plain classes, so a unit test needs no bus. Pass an explicit config though: when it is omitted, the base __init__ reads the plugin's section from the Configuration() singleton, which touches the on-disk config layers:

from my_module import MyCustomAudioTransformer

transformer = MyCustomAudioTransformer(config={"sample_rate": 16000})
audio_data, context = transformer.transform(b"\x00\x01")
assert audio_data == b"\x00\x01"

Verify discovery¶

After pip install -e .:

from ovos_plugin_manager.audio_transformers import find_audio_transformer_plugins

print(find_audio_transformer_plugins())
# {'my-custom-audio-transformer': <class 'my_module.MyCustomAudioTransformer'>}

Checklist before you publish¶

transform() accepts audio_data and returns (audio_data, context).
__init__ hardcodes the plugin name and forwards name, priority, config to super().__init__().
The entry-point group in pyproject.toml is opm.transformer.audio.
A unit test calls transform() directly, with no OVOS services running.
find_audio_transformer_plugins() discovers the installed plugin under the expected name.

Read next: TTS Transformers Related: Dialog Transformers · TTS Plugins · STT Plugins