OVOS Plugin Arena · How the numbers work

Methodology

Every ranking here is reproducible from public data: the vote log is a folder of GitHub issues, the datasets are pinned, and the whole board rebuilds deterministically. This page is the short version — the full methodology and the specification have the math and the normative rules.

Where ratings come from

Every ladder starts from auto-judged battles: two fighters' outputs on the same labelled sample, scored against the reference answer (exact match for intents, lower WER for STT, higher UTMOS for TTS) with no person involved. Each auto-judged battle is a pairwise comparison between two already-scored predictions, not an independent judge call — the scoring happens once per prediction at benchmark time, and every fighter pair sharing a sample is then compared on that same pair of scores. Those auto-battles are seeded into the rating at a fraction of a human vote's weight, so a large benchmark run bootstraps the board without drowning out real preference. From there, blind human votes cast in the battle page refine the same rating — the more human votes a matchup has, the more its rating reflects human preference rather than the benchmark seed. Every leaderboard states its own human-vote and auto-battle count, and the whole log is public: any board on this site can be rebuilt byte for byte by replaying the committed vote history, which is exactly what the replay-proof workflow does on every push.

Ratings: Bradley-Terry, not sequential ELO

Sequential ELO depends on the order votes arrive — replay the same votes in a different order and you get different numbers. The arena instead fits a Bradley-Terry model in one batch over the whole vote log (minorization-maximization), so the result is order-independent and the board is byte-identical on every rebuild. The legacy ELO column is kept for continuity but ranking is by the Bradley-Terry rating.

A convergence prior (one virtual tie against the mean per fighter) guarantees finite ratings even for a fighter that has only ever won or only ever lost, and connects otherwise-disconnected comparison graphs. Benchmark-seeded "auto" votes enter at a fraction of a human vote's weight.

Confidence, not false precision

Each rating carries a bootstrap 95% confidence interval (seeded, so it is replayable) computed by resampling the human votes. A board with too few human votes is flagged provisional rather than shown as a firm ranking. On the benchmark boards, any fighter whose interval overlaps the leader's is marked ≈ tied with #1 — the arena reports ranges, not decimal points it can't defend.

Seed-battle bias cap

Benchmarks bootstrap the boards before humans have voted, but a huge dataset must not drown out real votes. Auto-generated battles are only created from statistically significant metric gaps, and the total auto weight per fighter-pair is capped at roughly a handful of human votes regardless of dataset size. Seeds break the cold start; they never own the ranking.

Vote-fraud resistance — deterministic and replayable

Anti-abuse rules are pure functions of the committed vote log, so anyone can replay them: one counted vote per (voter, matchup); a per-voter daily cap; new accounts (recorded age at ingest, never re-fetched) carry zero weight; and heavily one-sided voters are down-weighted. Discarded votes are recorded, never deleted — the audit trail is public.

Replay the whole arena

Nothing here is a black box. Clone the repo, run assemble then tally, and you reproduce every board, badge, and confidence interval on this site. If you disagree with a number, you can check it.