Test Your Skill¶

In a nutshell

This page continues Your First Skill, and picks up right after it: the ovos-skill-my-first skill you just wrote. Here you'll add an automated test for it. It sends the skill the utterance "hello" and checks it replies correctly, without needing a microphone, speakers, or a running assistant. By the end you'll have a test you can run locally and in CI on every change. Did you land here without doing that tutorial first? See Testing Skills with ovoscope for the broader ovoscope reference, not tied to this one example skill.

Why test a skill end-to-end?¶

A skill can look correct: the code imports cleanly, the files are in the right place. It can still fail the moment a real user talks to it. The intent phrasing might not actually match, the dialog file might have a typo, or the entry point might not point at the right class.

An end-to-end (E2E) test catches these by running a small, in-process copy of OVOS, sending it a real utterance, and checking what comes back out. This is the same journey a spoken command takes, minus the microphone. ovoscope is the tool the OVOS project itself uses for this. Every skill accepted into the ecosystem is expected to ship at least one.

Step 1: Install ovoscope¶

Activate the same environment you installed the skill into in Your First Skill, the venv or container you ran pip install -e . in. ovoscope needs to be importable alongside the skill and OVOS itself, not just present somewhere on the machine.

Add it as a test dependency of the skill you built in Your First Skill:

pip install --pre ovoscope

The --pre flag is required, not optional. OVOS ships its current work as prereleases, and one of the packages ovoscope needs, ovos-spec-tools, has no stable release at all. Without the flag pip resolves an older combination that cannot run these tests.

For a real skill repository, list it under a [project.optional-dependencies] "test" extra in pyproject.toml instead of installing it loose, so pip install -e .[test] pulls in everything a contributor needs:

[project.optional-dependencies]
test = ["ovoscope"]

padacioso ships as part of ovos-workshop, a dependency of ovos-core itself, so no separate package is needed to test against it.

Step 2: Write the first `End2EndTest`¶

The Padacioso pipeline plugin needs no extra install

This test drives session.pipeline = ["ovos-padacioso-pipeline-plugin"]. Unlike Adapt or Padatious, that plugin ships with ovos-workshop and is always present once ovos-core is installed — there is nothing extra to pip install.

If the test runs but reports no spoken output at all, the pipeline plugin is not the likely cause; look at the intent file or the skill's dispatch handler instead.

Create test/test_hello.py next to your skill's pyproject.toml:

from ovos_bus_client.message import Message
from ovos_bus_client.session import Session
from ovoscope import End2EndTest

SKILL_ID = "my-first.youruser"


def test_hello_matches_and_speaks():
    session = Session("test-1")
    session.pipeline = ["ovos-padacioso-pipeline-plugin"]
    utterance = Message(
        "recognizer_loop:utterance",
        {"utterances": ["hello"], "lang": "en-US"},
        {"session": session.serialize(), "source": "A", "destination": "B"},
    )
    test = End2EndTest(
        skill_ids=[SKILL_ID],
        source_message=utterance,
        expected_messages=[],
        test_message_number=False,
    )
    # hello.dialog has three possible lines and OVOS picks one at random,
    # so assert the skill spoke one of the real candidates, not one fixed string.
    messages = test.execute()
    spoken = [m.data.get("utterance") for m in messages if m.msg_type == "ovos.utterance.speak"]
    assert spoken, "expected the skill to speak, got nothing"
    assert spoken[0] in {
        "Hello! Nice to meet you.",
        "Hi there!",
        "Hey — how can I help?",
    }

Why ovos-padacioso-pipeline-plugin, not Adapt?

Hello.intent is an exact-match intent file (one example phrase per line), the format shared by Padacioso, Padatious, and m2v. That's a different matcher from Adapt's keyword grammar. session.pipeline tells OVOS which intent engines to try, and in which order. It has to include the engine that actually understands your intent file, or the utterance is never matched. See Pipelines Overview for how the stages fit together.

skill_ids restricts which skill(s) ovoscope loads for the test, so you're only ever testing your own skill, not every skill installed on the machine.

Skip the boilerplate with the minicroft fixture

Once ovoscope is installed, it auto-registers a class-scoped minicroft pytest fixture through the pytest11 entry point. Put your tests in a class carrying a skill_ids = ["your-skill.your-name"] class attribute; each test_*(self, minicroft) method then receives a ready MiniCroft loaded with exactly those skills, started once per class, with no setUp/tearDown boilerplate. The fixture reads skill_ids from the test class, so a bare module-level test_*(minicroft) function silently gets a MiniCroft with zero skills loaded. The example above builds everything by hand so you can see what is actually happening; once you understand it, the fixture is the leaner way to write most tests.

Step 3: Run it¶

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest test/test_hello.py -v

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 avoids autoloading unrelated pytest plugins that some OVOS dependencies register. It fits the hand-built examples on this page, but it also disables ovoscope's own pytest11 plugin: with it set, the minicroft fixture from the tip above is not found unless you re-enable the plugin explicitly with -p ovoscope.pytest_plugin. A real run looks like this (trimmed of the framework's own deprecation-warning noise):

test/test_hello.py::test_hello_matches_and_speaks PASSED                [100%]

================== 1 passed, 563 warnings in 70.16s (0:01:10) ==================

First run is slow, and that's normal

The bulk of that ~70s is ovoscope spinning up a full in-process SkillManager and loading every intent-pipeline plugin installed on the machine (Adapt, Padacioso, Padatious, and any others you have), not just the one your test needs. On a machine with only the pipeline plugins your skill actually depends on, startup is much faster. On a shared box where several people run tests at once (a classroom, a CI runner), the runs compete for CPU — stagger them, or budget a couple of minutes per person.

Intent matched, handler dispatched, but nothing spoken?

If the capture times out waiting for ovos.utterance.handled even though the log shows ovos.intent.matched and your <skill_id>:<IntentName> dispatch message, the tutorial code is usually not the culprit — a crowded Python environment is. A venv with many co-installed skills and plugins can carry version skew or broken entry points that stall handler execution without a traceback. Scan the startup log for Failed to load plugin entry point errors, check pip list | grep ovos for mismatched ovos-core/ovos-workshop versions, and re-run the test in a fresh venv containing only your skill and its [test] extras before debugging the skill itself.

Step 4: Test the failure path¶

A test that only ever sends utterances your skill should match doesn't tell you much. Add a second test that sends something unrelated and checks the skill stays silent. This is what catches an intent file matching too eagerly:

# test/test_hello_nomatch.py
from ovos_bus_client.message import Message
from ovos_bus_client.session import Session
from ovoscope import End2EndTest

SKILL_ID = "my-first.youruser"


def test_unrelated_utterance_is_not_handled():
    """An utterance the intent files never taught the skill must NOT trigger it."""
    session = Session("test-2")
    session.pipeline = ["ovos-padacioso-pipeline-plugin"]
    utterance = Message(
        "recognizer_loop:utterance",
        {"utterances": ["what is the capital of france"], "lang": "en-US"},
        {"session": session.serialize(), "source": "A", "destination": "B"},
    )
    test = End2EndTest(
        skill_ids=[SKILL_ID],
        source_message=utterance,
        expected_messages=[],
        test_message_number=False,
    )
    messages = test.execute()
    spoken = [m.data.get("utterance") for m in messages if m.msg_type == "ovos.utterance.speak"]
    assert not spoken, f"skill should stay silent for an unrelated utterance, got: {spoken}"

test/test_hello_nomatch.py::test_unrelated_utterance_is_not_handled PASSED [100%]

================== 1 passed, 303 warnings in 72.87s (0:01:12) ==================

Testing an error dialog

If your skill calls self.speak_dialog("some_error") on a caught exception (a missing API key, a network failure, and so on), test that path the same way. Drive the skill into the failing condition and assert it spoke the error dialog's text instead of crashing silently or leaking a raw traceback to the user. ovos-skill-my-first has no such path. It can't fail, so there is nothing further to add here for this particular skill.

Step 5: Fixtures and the `ovoscope` CLI¶

ovoscope also ships a standalone CLI (ovoscope --help) for working with fixtures: recorded bus-message sequences you can replay without writing a pytest file:

Subcommand	What it does
`ovoscope record`	Capture a fixture: send one utterance to a MiniCroft instance and save the resulting messages to a JSON file.
`ovoscope run FIXTURE`	Replay a saved fixture and exit non-zero if it no longer matches.
`ovoscope diff A B`	Compare two fixture files.
`ovoscope validate FILE...`	Schema-validate one or more fixture files.
`ovoscope coverage`	Scan a workspace for which behaviors have E2E test coverage.
`ovoscope bus-coverage`	Run fixtures and report which bus message types were exercised.

A fixture is just the saved output of an End2EndTest. End2EndTest.from_message() runs the utterance through a real MiniCroft, captures every message that comes back, and hands you a test object whose .save(path) writes that captured sequence to JSON. This is what ovoscope record does internally:

# test/record_hello_fixture.py — one-off script, not a pytest file
from ovos_bus_client.message import Message
from ovos_bus_client.session import Session
from ovoscope import End2EndTest

session = Session("test-1")
session.pipeline = ["ovos-padacioso-pipeline-plugin"]
utterance = Message(
    "recognizer_loop:utterance",
    {"utterances": ["hello"], "lang": "en-US"},
    {"session": session.serialize(), "source": "A", "destination": "B"},
)
test = End2EndTest.from_message(
    utterance,
    ["my-first.youruser"],
    timeout=30,
    default_pipeline=["ovos-padacioso-pipeline-plugin"],
)
test.save("test/fixtures/hello.json")

Running it produces a fixture with the full 11-message sequence for this interaction: recognizer_loop:utterance, my-first.youruser.activate, ovos.intent.matched, ovos.intent.handler.start, the matched intent event (my-first.youruser:Hello), mycroft.skill.handler.start, ovos.utterance.speak, recognizer_loop:audio_output_start, mycroft.skill.handler.complete, ovos.intent.handler.complete, and ovos.utterance.handled.

$ python3 test/record_hello_fixture.py
saved OK

Validate its shape, then replay it:

ovoscope validate test/fixtures/hello.json

[validate] OK  test/fixtures/hello.json

ovoscope run test/fixtures/hello.json -v

A recorded fixture can be non-deterministic: watch for timestamps

Replaying the fixture above with ovoscope run reliably reports a mismatch. This is not because anything is actually broken. It happens because the session's active_handlers records a Unix timestamp (when the skill activated) at capture time, and that timestamp is different every time you re-run it:

[run] FAIL: ❌ message context mismatch for key 'session' - expected
'...activated_at': 1784823640.9039652...' | got '...activated_at': 1784823715.739504...'

Timestamps, request IDs, and anything else that legitimately changes between runs will do this to any fixture that captures full message context. Compare only the fields you actually care about instead of the whole context. Either pass assert_spoke()/execute() with a narrower expected_messages list in a pytest test (as in Steps 2-4), or use ovoscope diff with the default (context-skipping) comparison rather than --include-context when comparing fixtures.

ovoscope record from the command line

The ovoscope record subcommand does the same capture directly from the shell: ovoscope record --skill-id my-first.youruser --utterance "hello" --output test/fixtures/hello.json, without writing any Python. Both routes produce the same fixture format. Use whichever fits your workflow. The Python API is handy when you want to tweak the Session (language, pipeline list) before recording.

Multi-turn tests with a named session: the test owns the session, not the server¶

A test's Session almost always carries its own id, not the reserved "default" id. Under OVOS-SESSION-2 §2.2/§2.5, the orchestrator holds no state for a named session between utterances. The registry only ever stores the "default" session, so a named session_id never enters it, and there is nothing server-side to fetch back.

The test must act as the client would: build one Session object, put its current snapshot in every message's context["session"], and update that same object itself between turns from whatever the skill's own reply reported. Do not expect any server-side store to remember or merge a named session's state for you.

from ovos_bus_client.session import Session, SessionManager

session = Session("test-1")

# turn 1
bus.emit(make_utterance_message("book a table", session=session))
reply = wait_for_reply()  # your test's own capture/wait helper

# read what the skill actually changed off the reply's own session carrier,
# or off SessionManager.get(reply) for the message the skill handled
session = Session.deserialize(reply.context["session"])

# turn 2: reuse the session object you just updated
bus.emit(make_utterance_message("yes", session=session))

ovos-skill-naptime's end-to-end tests follow exactly this pattern: each test builds its own Session, stamps it onto the outgoing Message, and asserts against what the skill emitted back.

An observer on the skill's own FakeBus only hears one spelling¶

Subscribe to the canonical intent topic

A FakeBus models one bus connection, the same as a single real MessageBusClient. It carries the intent-topic bridge (the .intent-suffix compatibility described in Bus Namespace Migration), so within one connection only one spelling of a dispatch topic is delivered, never both. If your test attaches an observer to the same FakeBus the skill under test uses, that observer is not a second connection. It shares the guard with the skill's own subscriptions, so it must subscribe to the canonical topic, ovos_spec_tools.intent_topics.canonical_intent_topic(...), rather than assume it will see the legacy .intent-suffixed form as well. A separate real bus connection would hear both spellings; a shared FakeBus does not.

Step 6: Wire it into CI¶

Add a workflow that runs the test suite on every pull request:

name: Skill End-to-End Tests (ovoscope)

on:
  pull_request:
    branches: [dev, master, main]
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install skill + test deps
        run: pip install -e .[test]
      - name: Run ovoscope end-to-end tests
        run: |
          export PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
          pytest test/ -v

A shared workflow already exists

Rather than maintaining this yaml by hand in every skill repo, official OVOS skills call a shared, reusable ovoscope workflow instead.

jobs:
  ovoscope:
    uses: OpenVoiceOS/gh-automations/.github/workflows/ovoscope.yml@dev
    secrets: inherit
    with:
      python_version: '3.11'
      install_extras: 'test'
      test_path: 'test'

See GH-Automations Workflows for the full set of shared CI building blocks and what each input configures.