Skip to content

Reusable Workflow Reference: Skill Test Workflows

In a nutshell

This page documents the shared workflows that execute skill tests: ovoscope.yml for hand-written end-to-end skill tests, intent-case-tests.yml for the bulk generated intent-routing accuracy matrix, and tts-intelligibility.yml for synthesize-then-transcribe TTS scoring. For build and OVOS-plugin (OPM) checks, see Build Checks and Plugin, License, and Repo Checks. For skill-packaging checks, see Skill and Locale Checks. Start with the gh-automations overview for the big picture, or the full Workflow Reference index.

All reusable workflows are in .github/workflows/ and are called via:

uses: OpenVoiceOS/gh-automations/.github/workflows/<name>.yml@dev

Ref: Always use @dev.

Workflows on this page:

ovoscope.yml

Runs ovoscope end-to-end skill tests on a single Python version. Installs the skill with its test extras (which must include ovoscope), then executes pytest against the end-to-end test directory. Posts a 🔌 Skill Tests (ovoscope) section to the OVOS PR Checks comment.

Source: .github/workflows/ovoscope.yml

Inputs

Input Type Default Description
uv_prerelease string allow uv prerelease resolution mode (allow | if-necessary | explicit | disallow). Defaults to "allow": the OVOS ecosystem ships pre-1.0 alphas and relies on prerelease floor-pins resolving the way pip did.
pytest_workers string auto pytest-xdist worker count for the e2e suite. Each ovoscope test boots its own in-process MiniCroft and is independent, so they parallelize cleanly. "auto" = one worker per core; "0" runs serially (disable xdist).
runner string ubuntu-latest Runner label
python_version string 3.11 Python version to use
system_deps string "" Extra apt packages to install before testing (space-separated)
install_extras string test Extra dependencies used when installing the package; the extras must pull in ovoscope (e.g. ovoscope>=0.7.0 in [test]). Accepts a bare extras name ('test'), a bracketed list ('[dev,test]'), a full target ('.[test]'), or raw pip arguments ('-r requirements/test.txt').
pre_install_pip string "" Optional space-separated pip requirement specs to install BEFORE the package install step. Use to override transitive deps with git URLs (e.g. to test against an unreleased sibling package). Mirrors the build-tests workflow input.
post_install_pip string "" Optional space-separated pip requirement specs installed AFTER the package + test-extras install, with --force-reinstall --no-deps so the dependency resolver cannot revert them. Use to pin an unreleased prerelease (e.g. a sibling at the same version as a published wheel) that the test-extras resolution would otherwise downgrade.
test_path string test/end2end/ Path passed to pytest — should point at the end2end directory
require_adapt boolean false Auto-install ovos-adapt-pipeline-plugin and fail CI if installation fails. When false, Adapt pipeline tests are skipped if the plugin is absent.
require_padatious boolean false Auto-install ovos-padatious (PyPI name) and fail CI if installation fails. When false, Padatious pipeline tests are skipped if the plugin is absent.
require_m2v boolean false Auto-install ovos-m2v-pipeline and fail CI if installation fails. When false, M2V pipeline tests are skipped if the plugin is absent.
require_nebulento boolean false Auto-install nebulento and fail CI if installation fails. When false, nebulento pipeline tests are skipped if the plugin is absent.
require_palavreado boolean false Auto-install palavreado and fail CI if installation fails. When false, palavreado pipeline tests are skipped if the plugin is absent.
pre_release boolean false Install ovoscope and every required pipeline plugin at its latest PyPI prerelease instead of a floor-pinned version. Use to validate against unreleased changes in the framework or any engine. Affects ovoscope itself (always upgraded to the latest PyPI prerelease when this flag is set) and the engines gated by require_adapt / require_padatious / require_m2v / require_nebulento / require_palavreado.
bus_coverage boolean true Enable bus-level message coverage tracking
bus_coverage_include string "" Regex pattern of skill_ids to include in bus coverage report
bus_coverage_exclude string ^Thread-|^intents$|^skills$|^__core__$ Regex pattern of skill_ids to exclude from bus coverage report
pr_comment boolean true Post a '🔌 Skill Tests (ovoscope)' section in the OVOS PR Checks comment. Only runs on pull_request events.

Jobs

Job Description
ovoscope Installs system deps, then installs the package with test extras plus ovoscope and pytest-json-report. Runs a pipeline availability check (fails fast if require_* inputs are true and the plugin is absent). Executes pytest with --json-report, formats the results, and posts the PR comment section.

Steps

Step Description
Checkout Checks out the calling repo
Checkout gh-automations scripts Checks out OpenVoiceOS/gh-automations@dev into _gh_automations/ (PR events only)
Setup Python actions/setup-python@v6
Install System Dependencies apt-get install the system_deps list (skipped if empty)
Install Package with Test Extras pip install ".[test]" (or the configured extras) plus pytest pytest-json-report ovoscope
Check required pipeline availability Inline Python reads opm.pipeline entry points and exits 1 if any require_* pipeline is absent
Run ovoscope tests pytest --json-report with continue-on-error: true so the PR comment step always runs
Format ovoscope section for PR comment Inline Python reads the JSON report and generates ovoscope-section.md grouped by test class
Post ovoscope section to PR comment Calls update_pr_comment.py with --section-id ovoscope
Fail job if tests failed Re-raises the pytest failure after the PR comment is posted

PR comment content

✅ 9/9 passed

✅ **TestConfuciusAdaptEN** — 5/5
✅ **TestConfuciusPadaciosaEN** — 2/2
✅ **TestConfuciusFixtures** — 2/2

On failure, failing classes expand to a per-test table with longrepr for the first 3 failures.

Typical usage

name: Ovoscope End-to-End Tests
on:
  pull_request:
    branches: [dev]
  workflow_dispatch:

jobs:
  ovoscope:
    uses: OpenVoiceOS/gh-automations/.github/workflows/ovoscope.yml@dev
    secrets: inherit
    with:
      test_path: "test/end2end/"
      require_adapt: true

To require Padatious:

    with:
      test_path: "test/end2end/"
      require_adapt: true
      require_padatious: true

Notes

  • The require_* inputs trigger a pre-test pipeline availability check. If the required plugin is absent the job fails immediately with a clear error message, without running any tests.

  • The pipeline check reads the opm.pipeline entry point group using importlib.metadata, no import of the plugin itself is required.

  • PADACIOSO_PIPELINE (pure Python padacioso) is always available via ovos-workshop. There is no require_padacioso input.

  • Set require_adapt: true in skill repos that test Adapt intents. This makes CI fail explicitly if the Adapt plugin is missing from [test] deps, rather than silently skipping those tests.

intent-case-tests.yml

Runs the file-based ovoscope intent-routing accuracy matrix. Each <IntentName>.intent.test / no_match.test file under cases/<lang>/ is run through every pipeline, producing one test per (pipeline, lang, utterance). Sharded by language (one matrix job per lang) so large suites don't time out. Posts a 🎯 Intent-Case Accuracy section.

Use alongside ovoscope.yml: ovoscope.yml for hand-written E2E tests and bus coverage, intent-case-tests.yml for the bulk generated intent-routing matrix.

Source: .github/workflows/intent-case-tests.yml

Inputs

Input Type Default Description
uv_prerelease string allow uv prerelease resolution mode (allow | if-necessary | explicit | disallow). Defaults to "allow": the OVOS ecosystem ships pre-1.0 alphas and relies on prerelease floor-pins resolving the way pip did.
runner string ubuntu-latest
python_version string 3.11
system_deps string "" Extra apt packages to install before testing (space-separated)
install_extras string test Extra dependencies used when installing the package; must pull in ovoscope. Accepts a bare extras name ('test'), a bracketed list ('[dev,test]'), a full target ('.[test]'), or raw pip arguments ('-r requirements/test.txt').
test_path string test/end2end/ Path passed to pytest. Default test/end2end/ collects the whole end2end suite; ovoscope's intent-case auto-discovery picks up the shim file (ovoscope_intent_cases = {...}) inside it.
require_adapt boolean false
require_padatious boolean false
require_m2v boolean false
require_nebulento boolean false
require_palavreado boolean false
pre_release boolean false Install ovoscope and every required pipeline plugin from its GitHub dev branch instead of PyPI.
pr_comment boolean true Post a '🎯 Intent-Case Accuracy' section per shard in the OVOS PR Checks comment (only on pull_request).
langs string ["all"] JSON array of language tags. One matrix job per lang. Each shard passes -k '<slug>' to pytest (slug = lang.replace('-', '_')) so only tests whose nodeid contains that slug run there. Pass the skill's locale list verbatim, e.g. '["en-US", "de-DE", "pt-PT"]'. The default ["all"] runs the entire suite in a single job.
accuracy_baseline string "" Path (relative to the skill repo root) of a previously-saved intent-accuracy report JSON. When supplied, the run fails on regressed (was-pass → now-fail) cases.
accuracy_tolerant boolean false Downgrade individual case failures to xfail. Only the aggregate gate (accuracy_min / regression list) can block the session.
accuracy_min string "" Minimum overall pass rate (0.0-1.0). Fails the shard when the observed rate is below this value.

tts-intelligibility.yml

End-to-end TTS intelligibility scoring. Synthesises speech with the TTS plugin under test, transcribes it back with a reference STT (faster-whisper tiny), and scores the round-trip with WER/CER. Posts a 🗣️ TTS Intelligibility section. Gates by default, the job fails when mean WER exceeds max_wer or the test errors. Set warn_only: true for engines/langs still being tuned.

Source: .github/workflows/tts-intelligibility.yml

Inputs

Input Type Default Description
uv_prerelease string allow uv prerelease resolution mode (allow | if-necessary | explicit | disallow). Defaults to "allow": the OVOS ecosystem ships pre-1.0 alphas and relies on prerelease floor-pins resolving the way pip did.
runner string ubuntu-latest Runner label
python_version string 3.11 Python version to use
system_deps string "" Extra apt packages to install before testing (space-separated), e.g. 'espeak-ng sox'. Engines backed by a system binary need this.
install_extras string test Extra dependencies used when installing the package; the extras should pull in ovoscope[tts]. Accepts a bare extras name ('test'), a bracketed list ('[dev,test]'), a full target ('.[test]'), or raw pip arguments ('-r requirements/test.txt').
test_path string test/end2end/test_tts_intelligibility.py Path passed to pytest — the intelligibility test file
max_wer string 0.5 Intelligibility gate (exported as TTS_MAX_WER): the test fails when the mean word-error-rate of the STT round-trip exceeds this. Default 0.5 — more than half the words wrong is treated as unintelligible. Raise it per-plugin for engines/voices/low-resource langs the reference STT transcribes weakly; lower it to tighten the bar.
pre_release boolean false Install ovoscope[tts] from its GitHub dev branch instead of PyPI. Use to validate against unreleased ovoscope changes.
pr_comment boolean true Post a '🗣️ TTS Intelligibility' section in the OVOS PR Checks comment. Only runs on pull_request events.
warn_only boolean false When false (default), the check GATES: an unintelligible round-trip (mean WER > max_wer) or a test-process error fails the PR. Set true to downgrade to report-only — the score is still posted to the PR comment but the check never fails — for engines/langs still being tuned.

Read next: Quality Workflows Related: gh-automations Overview · Build Checks · Plugin, License, and Repo Checks · Skill and Locale Checks · Workflow Reference