@keeb/rote-classifier

Decide whether an issue is ROTE: safe to run plan → implement → PR → auto-merge with no human review. Anything else is HUMAN.

The classifier is a rubric and a gate:

• Rubric. Jev (TypeSafe System One) answers 15 yes/no questions about the issue through an @swamp/typesafe-ai definition. Each answer is a probability: is it docs-only, does it need a product decision, does it touch auth or billing code, a schema, infrastructure…
• Gate. A fixed rule turns the probabilities into a verdict and names every condition that failed. It fails closed: an unanswered question blocks.

Where the issues come from does not matter. classify takes plain {id, title, body, type?} records, so issues from the swamp-club Lab, GitHub or Linear all go in the same way.

Calibration

The bundled rubric is v6 with the v6 docs-branch gate (rubric/). They were calibrated on 244 hand-labelled swamp-club Lab issues against jev-latest (jev-1.13.0):

Set	Precision	Recall
Tune (130)	1.00	0.74
Holdout (56, run once)	0.97	0.97
Open (58, human prior)	0.88	0.47

A false ROTE costs far more than a missed one, so the gate is tuned for precision. Every false ROTE seen was a borderline issue; no clearly-not-rote issue was ever flagged. By design only documentation issues can pass the gate, so code fixes always go to a person.

The issue source is pluggable, but the rubric is not neutral: several risk questions name swamp-club's own stack (ClickHouse, MongoDB, XP and the ledger, the telemetry service, the discord bot). For another codebase, re-word those questions for its stack and re-check the gate with the questions and gate overrides before trusting a ROTE.

Each verdict records the Jev checkpoint that answered. The gate was fitted on jev-1.13.0; a re-run of the 58 open issues on jev-1.14.0 gave the same precision and recall and the same single false ROTE, but check a new checkpoint the same way before relying on it.

Setup

swamp extension pull @keeb/rote-classifier   # pulls @swamp/typesafe-ai too

# Jev, with the key in a vault
swamp vault create local_encryption typesafe
swamp vault put typesafe TYPESAFE_API_KEY
swamp model create @swamp/typesafe-ai jev \
  --global-arg 'apiKey=${{ vault.get("typesafe", "TYPESAFE_API_KEY") }}'

# The classifier, pointed at that definition
swamp model create @keeb/rote-classifier rote --global-arg jev=jev

Classify

swamp model method run rote classify --input-file issues.json

{
  "issues": [
    {
      "id": 2036,
      "type": "bug",
      "title": "Docs: the env example is wrong",
      "body": "…"
    },
    { "id": "owner/repo#12", "title": "Add a cache for scores", "body": "…" }
  ]
}

Up to 100 issues per call; classify asks Jev once per issue through runModel, and swamp allows 100 such calls per execution. Larger sets go in batches.

In a workflow, wire the issues in with CEL from wherever another model stored them:

- name: classify
  task:
    type: model_method
    modelIdOrName: rote
    methodName: classify
    inputs:
      issues: ${{ data.latest("tracker", "open").attributes.issues.map(i, {"id": i.number, "title": i.title, "body": i.body}) }}

Output

Data	What it holds
verdict-<id>	verdict (ROTE, HUMAN, UNCLASSIFIED), blockers (e.g. needs_decision=0.71>0.3), every answers.<question>, the Jev checkpoint, rubric and gate hashes
summary	counts and the rote, human and unclassified id lists for the run (name it with --input name=…)

# Only auto-merge what the gate passed
${{ data.latest("rote", "verdict-2036").attributes.verdict == "ROTE" }}

An issue Jev could not be asked about is UNCLASSIFIED with the error, and the rest of the batch still runs. A run where nothing could be asked fails. Jev's raw answers stay on the jev definition as evaluation-rote-<id>.

Calibration overrides

questions and gate replace the bundled rubric and gate for one call, for trying a new question or threshold. The stored verdicts record rubric.label custom and both hashes, so runs stay comparable:

swamp model method run rote classify --input-file issues.json \
  --input 'gate={"type":"rule","version":"strict","min":{"docs_only":0.9},"max":{"needs_decision":0.2}}'

A gate is either {"type": "rule", "max": {…}, "min": {…}, "anyMin": {…}} or {"type": "logistic", "intercept": …, "weights": {…}, "threshold": …}.