ScoreExplainer, also embedded on Console → Scoring).
The formula
>= 0.5. In practice a merged PR (1.0) clears the bar; a closed-unmerged or reverted PR (0.0) does not.
Spending less only raises the score if accepted sessions hold; if accepted sessions drop as spend drops, the score does not improve. There is no cheapness-only path to a higher number, because cost only ever appears in the denominator.
What “accepted” means
Quality comes from a GitHub PR outcome, not a manual grade — there is no “accept” button on a receipt, and no LLM-as-judge step:- A merged pull request scores
1.0. - A closed, unmerged pull request scores
0.0. - A pull request that is later reverted re-scores back down to
0.0.
Wardin-Session-Id: footer line in the PR body, matching the session id sent via X-Wardin-Session-Id on the requests that did the work — parsed straight from the GitHub pull_request webhook payload, no GitHub API calls in the hot path. It’s opt-in per team (the repo needs a connected GitHub webhook) and per PR (the footer has to be present) — a PR without either produces no signal, never a guessed one. See Sessions as a quality signal for the full mechanism and the Claude Code guide for a copy-paste CLAUDE.md snippet.
Manual per-receipt grading was deliberately removed: in a passive, high-volume system nobody grades individual receipts one at a time, and the person being measured can’t credibly grade their own output. A peer-reviewed, branch-protected merge event is the live source instead — someone other than the author approved the work, which is audit-defensible in a way a self-click never is.
Where each diagnostic number comes from
OUTCOMES also shows a row of diagnostic tiles and a per-engineer scorecard. These are informational only — never blended into the headline ratio (a raw efficiency number is never shipped without a paired quality signal).| Signal | Source | Rail stage |
|---|---|---|
| Cost / session | Session-grouped spend from gateway usage events | LEDGER |
| Economy-tier mix | Share of requests served by haiku/mini/flash-class models | ROUTE |
| Cache-assist rate | Exact + semantic cache hit share of requests | Caching |
| Policy / budget blocks | Guardrail denials (model allowlist, prompt injection) and rate/budget limits — PII is redacted, not blocked | Policies |
| Quality accept-rate | GitHub PR outcome for the session, for teams that opt in | Evidence layer |
Coverage-gated ranking
A ranked leaderboard only tells the truth once there’s enough quality evidence behind it. The earned-mode ranked leaderboard unlocks once>= 60% of active interactive keys carry >= 5 legitimate (deduplicated, non-self) quality signals in the trailing 30-day window.
Below that threshold, OUTCOMES shows only the per-engineer scorecard — a set of independently defensible facts (cost/session, budget adherence, policy blocks, cache-assist rate, quality accept-rate, economy-tier mix) with no composite score and no rank, rather than a ranking built on too little quality evidence to be trustworthy.
Quality figures backed by fewer than 3 signals are flagged provisional rather than shown as a settled number.
Anti-gaming rules
- Quality is not self-reported. The signal is a peer-reviewed, branch-protected merge event, not a button the developer being measured can click.
- CI/headless keys are excluded from every score, ratio, and ranking. They have their own budget pool (Sessions), not a developer’s — mixing the two would let automated volume dilute a human quality signal.
- Raw token counts are never scored. Every number in the pipeline is a session, a dollar, or a quality signal — never a token or line-of-code count, which are gameable once AI-assisted work is widespread.