Testing Skills with `ovoscope`¶

In a nutshell

ovoscope is the official tool for testing OpenVoiceOS skills and core parts. It runs a small, fast, pretend version of the whole assistant in one process. This lets you confirm a request is understood and answered correctly without real hardware. "End-to-end" just means it checks the whole journey, from what the user said to what the assistant did. This page gets you started. The full, always-current reference lives in the ovoscope repo's docs/. See the Glossary.

This page is a guide; the reference is in the repo

ovoscope is actively developed, so its detailed per-harness reference is kept in the tool's own docs/ rather than copied here (which would drift out of sync). This page gives you the concept and a working first test, then links to the authoritative reference for each harness.

ovoscope provides a lightweight, in-process environment for running End-to-End (E2E) tests without a full system installation. Every official OVOS skill is required to pass ovoscope E2E tests.

Why E2E testing?¶

Traditional unit tests miss integration issues (for example, a skill that loads fine but fails to match a Padacioso intent). ovoscope solves this by running a MiniCroft instance, a real in-process SkillManager:

Real intent matching: uses the actual Adapt and Padacioso engines.
Bus-level verification: asserts the correct speak / gui.page.show messages are emitted.
No hardware: uses a FakeBus and mocked audio/hardware layers.
CI ready: designed to run in GitHub Actions on every pull request.

Padatious is opt-in in CI

A skill-intent test suite does not boot Padatious by default. The shared ovoscope CI workflow installs plain ovos-core plus Padacioso, and only pulls in ovos-padatious when the caller sets require_padatious: true — see Test Your Skill. A suite that genuinely needs Padatious opts in with that input, plus the matching plugin in its own [test] extras.

Your first test¶

End2EndTest is declarative: give it the skill(s) to load, the utterance to send, and the messages you expect back. The simplest assertion is assert_spoke(). It runs the interaction and checks the skill spoke a given line, without spelling out the full message sequence:

from ovos_bus_client.message import Message
from ovos_bus_client.session import Session
from ovoscope import End2EndTest

SKILL_ID = "ovos-skill-hello-world.openvoiceos"

def test_hello_world():
    session = Session("test-1")
    session.pipeline = ["ovos-padacioso-pipeline-plugin"]
    utterance = Message(
        "recognizer_loop:utterance",
        {"utterances": ["hello world"], "lang": "en-US"},
        {"session": session.serialize(), "source": "A", "destination": "B"},
    )
    test = End2EndTest(
        skill_ids=[SKILL_ID],
        source_message=utterance,
        expected_messages=[],          # we are not checking the full sequence
        test_message_number=False,     # so don't assert on the message count
    )
    test.assert_spoke("Hello world")   # runs execute() internally, then scans for speak

assert_spoke(text, lang="en-US", timeout=30) runs execute() and then raises AssertionError if no speak message with that exact utterance (and lang) was emitted. Because it still runs the full execute(), leave expected_messages empty and set test_message_number=False. Otherwise execute() first asserts the received-message count equals the (empty) expected list. It then fails before the speak is ever checked.

For full sequence assertions — message types, ordering, routing, and session state — populate expected_messages and call test.execute() directly (see the usage guide).

ovoscope also ships ready-made pipeline-stage lists, so you do not have to hand-write session.pipeline = [...] for common cases: ADAPT_PIPELINE, PADACIOSO_PIPELINE, FALLBACK_PIPELINE, PERSONA_PIPELINE, and DEFAULT_TEST_PIPELINE (a deterministic mix that deliberately excludes persona/Ollama/OCP/m2v plugins). Import them from ovoscope and assign to session.pipeline. See end2end-test.md for the full list.

Testing skills still in development, and non-utterance triggers¶

Pass extra_skills={skill_id: SkillClass} to MiniCroft to load a skill class directly, with no PyPI entry point required. This is useful for testing a skill before it is packaged.

To drive a non-utterance handler — a GUI event, a timer firing, an API call — call MiniCroft.inject_message() instead of sending an utterance through the pipeline.

What `ovoscope` does not do¶

ovoscope tests at the recognizer_loop:utterance level and in-process. It does not:

open a real WebSocket bus
run the PHAL or audio services
render the GUI
perform real TTS

Keep this in mind when a test needs one of these — it needs the real stack (or a dedicated harness), not ovoscope.

The harnesses (full reference in the repo)¶

ovoscope ships a harness per subsystem. Each is documented in the tool's docs/:

Harness / topic	What it tests	Reference
`MiniCroft`	The in-process core that loads your skills	minicroft.md
`End2EndTest`	Declarative single/multi-turn interactions	end2end-test.md · usage-guide.md
`CaptureSession`	Recording bus messages for assertion/replay	capture-session.md
Pipeline testing	Intent-pipeline matching	pipeline.md
OCP testing	Media ("play …") skills	ocp.md · media-testing.md
Audio service testing	TTS / sound playback lifecycle	audio-testing.md
Listener testing	The speech/listener service	listener.md · voice-loop.md
`WakeWordProbe`	Streams a single audio clip through a real `HotWordEngine` with listener-style silence priming, for per-clip wake-word accuracy checks	`ovoscope/wakeword_probe.py`
PHAL testing	Hardware-abstraction plugins	phal.md
GUI testing	`gui.page.show` and GUI state	gui-testing.md
Pydantic integration	Typed message assertions	pydantic-integration.md
Bus coverage	Which message types your test exercised	bus-coverage.md
CLI	The `ovoscope` command-line runner	cli.md
`ovoscope-setup`	Companion command that installs the ovoscope skill for AI coding assistants (`--claude`, `--gemini`)	repo docs

The low-level FakeBus it builds on lives in ovos-utils (ovos_utils.fakebus).

`MiniCroft` training timeouts¶

get_minicroft() waits for mycroft.skills.trained to go quiet before returning. If training never completes within OVOSCOPE_TRAINED_TIMEOUT seconds, it raises RuntimeError. That timeout defaults to 180s when the CI environment variable is set, and 5s otherwise. Override either default with the env var. Fleet CI runs under coverage instrumentation on throttled 2-core runners, and a large single-skill intent set has exceeded 60s in the field.

Set pytest-timeout to at least the trained-wait ceiling plus 120s on any test suite that uses get_minicroft. For single-language suites on the 180s CI default, use 300s or more. For multilingual suites, set pytest-timeout to at least max_wait + 120s. A pytest-timeout at or below the ceiling kills setup mid-wait, and the failure looks like a boot failure instead of a timeout. This applies to the global timeout in pytest.ini or pyproject.toml too, not just a per-test @pytest.mark.timeout: a low global floor kills setUpClass before get_minicroft even returns, regardless of any per-test marker. Full detail: minicroft.md.

Running in CI¶

Add the shared ovoscope workflow to your skill repo so the tests run on every pull request — see CI Integration and the manual's gh-automations page for the OVOS CI conventions.

Source code & full reference: OpenVoiceOS/ovoscope. See its docs/.

Read next: Skill Development Overview Related: Test Your Skill · Fallback Skill · Common Query Framework · Developer FAQ

Testing Skills with ovoscope¶