What the index measures

For each behaviour, three strata, each on a 0–4 scale:

• Coverage -- what each lab's own specification declares about the behaviour, cited verbatim.
• Instruments -- what public evaluations of the behaviour exist. A fact about the field, not about any lab.
• Adherence -- what those evaluations report per lab, as historical per-paper snapshots.

The unit of the index is the behaviour (e.g. § 1 No sycophancy). Each behaviour is operationalized as facets: concrete eval questions with a setup, a metric, and a pass condition. Evidence attaches at the facet level; a behaviour with no eval-covered facets is an evidence gap, which is itself an index finding.

Evidence enters through a behaviour sweep: a staged pipeline run one behaviour at a time. Every stage produces a committed artifact and stops at a gate -- a checklist a human verifies and signs before the next stage may start.

Stage 1Discover

Find every pre-existing evaluation of the behaviour. Discovery is complete, not selective.

• A seed list is written down before searching: candidates named in the behaviour's definition, prior sweeps' watchlists, anything already known. Seeds are then verified like any other candidate, not trusted.
• Two to three research agents search in parallel: one per known-candidate cluster, plus one broad sweep of recent literature (behaviour terms, venue programs, citation trails from the seeds in both directions).
• Everything found enters the candidate register, including instruments that will obviously be excluded. Exclusion is a documented decision at stage 2, never a silent omission during search.
• Each candidate gets a dossier: full citation; every URL fetched live during the sweep with its status recorded (dead or gated links are marked, never dropped); what it measures; a mapping onto the behaviour's facets; the methodology facts the rubric needs; the instrument's last-activity date; per-model results with exact model versions; limitations and critiques from the literature.
• Every fact in a dossier is labelled with where it comes from: we opened the source and checked it ourselves, it is the paper's own claim (which we did not re-check), or a third party reports it. Nothing is included just because the model “remembers” it -- a claim with no source fetched during the sweep is marked unverified.
• A search log records each agent's scope, queries, dates, and zero-result probes, so coverage can be judged from the log alone.

Gate 1 -- the evidence base is real and complete. The human spot-checks two random candidates against their primary sources.

Stage 2Curate

Decide what counts as index evidence. Every candidate receives exactly one final disposition: curated, rejected, watchlist (would qualify if a named condition is met), context (informs findings but is not index evidence), or port (repackages another instrument's data).

• Dispositions are assigned against pre-registered exclusion criteria, applied as written. A candidate is excluded when it:
  • measures a different construct than the behaviour;
  • contributes no new evidence (a port of another instrument's data);
  • reports a lab's own numbers from an internal eval that cannot be inspected;
  • falls below the rigor floor;
  • is too small to bear weight;
  • has been dormant for more than two years;
  • is a methodology tool rather than a benchmark;
  • is off-domain for a general index.
  A candidate that fits none of these is a taxonomy gap raised to the human, not a stretched criterion.
• How many are curated is an outcome, not a quota: every candidate that fits the behaviour's facets and clears the quality bar is kept, and the set stops growing when the next-best candidate would add no new facet coverage, no independent corroboration, and no methodological diversity. That saturation judgment is recorded for the first candidate left out.
• Partial use: when a curated eval measures a construct broader than the behaviour, the subtests actually in scope (its “slices”) are recorded, and later stages may use only those.
• Lab-built evals: what matters is replicability, not authorship. A lab-built eval that is public and independently re-runnable is eligible index evidence, scored like any other. Numbers from internal evals that cannot be inspected (e.g. model-card claims) are never index evidence -- but they are kept, because a gap between a lab's own numbers and public, replicable measurement is itself a finding.
• Leave-outs are documented with a criterion code and a one-line reason -- never deleted. The register is the transparency record of what was used and what was left out, and why.

Gate 2 -- the sweep's editorial decision point. The human reviews the curated list and the full leave-out list, confirms or overrides every disposition, and the signed decision line is quoted in the published write-up.

Stage 3Score

Each curated eval is scored 0–4 on three dimensions taken from RAND's criteria for rigorous model evaluations (Paskov et al. 2025): internal validity, external validity, and reproducibility.

• A score with no named checklist items is invalid. Each dimension score is justified by listing the rubric items met and unmet, in a 15-item verdict matrix per eval.
• Verdict vocabulary: met / partial / not met / not reported (the paper is silent) / not verified (we did not check). The last two are different claims and are never collapsed; everything unverified goes on a known-unknowns list with what checking it would take.
• Every verdict carries an evidence tier; every dimension a confidence level, with the reason whenever it is not high.
• Adherence extraction: where a paper reports Anthropic or OpenAI results, every reported model's result is recorded in the sweep record -- the in-scope slice only, each with its underlying number, exact model version (or “unpinned”, stated as such), and date. Older-model results are kept and dated, never discarded: the date label is what stops a 2023 number being read as current.
• The single per-lab band shown on the index is the band of the most recent model version that eval measured for that lab, mapped to 0–4 -- never an average across model generations. The full per-model table stays in the sweep record so the band can be re-derived. (A per-model view on the site is planned.)
• Numbers are never invented: a missing cell is null with a reason. All adherence is labeled historical -- a per-paper snapshot, not a current-model verdict.

Gate 3 -- scores are auditable. The human picks one eval × dimension and confirms the named checklist items support the score.

Stage 4Spec coverage · parallel track

Independently of stages 1–3, extract what each lab's specification says about the behaviour. The ground truth is the latest published version of each spec -- the Claude constitution and the OpenAI Model Spec -- mirrored into the repository from the labs' own published sources and confirmed current at the start of the sweep.

• A term list is built before searching -- the behaviour's own words, synonyms, antonym-phrases, and spec-register phrasings -- and documented in full in the sweep's public spec-coverage record, including the terms that returned nothing: empty probes are part of the evidence that the sweep was exhaustive.
• A search hit is a starting point, not the passage: the whole surrounding section is read, because a spec often states the actual rule a few sentences away from the word that matched.
• Every quote is exact, never edited or abridged, and is pinned to the spec version it came from -- each is re-checked mechanically against the spec text rather than copied by hand.
• Each excerpt carries a one-line role; boundary passages are kept but marked adjacent, with the reason they sit outside the core construct.
• Each spec receives a verdict (covered / partial / not in spec) and a depth score 0–4 with a rationale naming what is present and what is missing.

Gate 4 -- quotes are mechanical, not remembered. Every stored quote is re-checked against the spec text in a scripted loop, with zero mismatches; the human spot-reads the set.

Stage 5Publish · internal

Nothing new is decided here. Publishing transcribes the gate-approved artifacts to the sweep's internal surfaces: the repository (a canonical write-up plus machine-readable data files) and internal analysis pages. This is preliminary, internal publication -- the public site is not updated at this stage. Divergence between a surface and its artifact is a bug; if transcription surfaces an error, the stage artifact is fixed first and the fix logged at its gate.

• The write-up includes the method, the rubric scores with their item-level justifications, the spec excerpts, the rejected-candidates table, the adherence summary, and a whole-sweep epistemic status and provenance section: evidence tiers, known unknowns, and the conflict-of-interest note.
• The published assessment block claims exactly what the gate log supports -- a gate spot-audit is not an item-by-item human review, and is not described as one.

Gate 5 -- every surface is faithful to the artifacts. Data files are validated, scores are diffed across surfaces, and written pages are re-fetched and checked.

Stage 6Verify, then release

A fresh session that did not run the sweep audits it end-to-end -- an auditor that watched the sweep shares its blind spots.

• Register accounting: every candidate found has exactly one disposition and is accounted for on every surface it should have reached.
• Score identity: rubric scores and adherence bands are extracted mechanically from all four records (write-up, data files, analysis pages, and the site's source data) and diffed -- not eyeballed.
• Every source link is re-fetched live; every spec quote is re-resolved against the pinned spec copy.
• Random dossier facts and adherence numbers are traced back to their primary sources.
• Discrepancies are fixed in the owning stage's artifact and re-propagated; the audit report keeps both the finding and its resolution. A clean report that hides a fixed discrepancy is a false record.

Gate 6 -- the sweep is complete. The human signs the final line, and only then is the public site deployed: nothing reaches this page unverified. After Gate 6, every candidate is accounted for, every number traces to a source, every quote to its exact spec text, and every review step to a dated sign-off.

The gates

• Every stage ends by rendering its checklist with pointed evidence -- fetch logs, command output, links into the artifact -- never bare checkmarks. Then it stops.
• Corrections loop within the stage; the next stage may not begin, even speculatively, until the gate is signed. (The one exception: stage 4 runs parallel to stages 1–3.)
• Sign-offs, corrections, and accepted open items are logged per gate. An open item accepted at a gate is carried into the write-up's known-unknowns section -- accepted never means dropped.

Provenance & honesty

• Sweeps are run by Claude, an Anthropic model, directed by Andrés Cotton. An index that scores Anthropic models, produced with an Anthropic model, is a conflict of interest; it is declared on every eval record rather than hidden.
• “Human review” here means the gate record -- dated sign-offs and spot-audits -- not an item-by-item review. No surface claims more than the gate log supports.
• All adherence numbers are historical per-paper snapshots, each carrying its model version and date. None are current-model verdicts.
• The pipeline itself is public: it is codified as versioned procedure files in the repository (.claude/skills/), alongside every register, dossier, and gate log. The pipeline was established by the No sycophancy sweep of 2026-07-12 and split into staged procedure files on 2026-07-13; that first sweep predates parts of the current layout and its record says so.

Sources

• Paskov et al. (2025), criteria for rigorous model evaluations, RAND -- rand.org/pubs/perspectives/PEA3971-1. Source of the three rubric dimensions and the checklist items behind each 0–4 score.
• Anthropic, Claude's constitution -- anthropic.com/news/claude-new-constitution. Latest published version, constitution@2026-01-20, mirrored from Anthropic's own published source.
• OpenAI, Model Spec -- model-spec.openai.com/2025-12-18. Latest published version, model-spec@2025-12-18, mirrored from OpenAI's own published source.
• Working records: registers, dossiers, curation memos, scores, gate logs, and the citation convention (specs/CITATION.md) -- github.com/AndresCotton/ai-character-index.
• The curated evaluations themselves are cited per behaviour, on each clause's record on the index.