Pre-review and what blocks prod

Pre-review and what blocks prod

You build capstone, a pre-review gate for pull requests: it loads the repository’s path-scoped review rules, runs a read-only agent over each PR with a structured findings schema, cleans the findings, applies the patch to a copy and runs the tests, ranks what blocks each PR from repository signals, measures the reviewer against labels, and writes the report a team lead reads before a human reviews anything.

The target is fixtures/go-ts-mini (see its README.md): a Go translation jobs service with a TypeScript client, a proto contract, numbered SQL migrations, a JSON changelog, five rules under .claude/rules/ and twelve pull requests under prs/pr-01 … prs/pr-12, each a patch.diff and a meta.json. The answers for those PRs live in this exercise’s tests/ folder: LABELS.yaml (the planted issues, blockers, verify outcome and decision per PR) and FAKEREVIEW.yaml (a recorded reviewer, wrong in the ways real reviewers are wrong).

capstone/types.py is given and shared by every stage: Finding, Label, Rule, PR, ReviewResult, Applied, VerifyResult, Reconciled, Blocker, Ranked, Matches, Score, MetricsReport and the enums CATEGORIES, SEVERITIES, BLOCKER_KINDS. Do not change it; write the stage modules against it.

How to run

Run every command below in a terminal at the root of your clone of this repository, after a one-time uv sync there; uv run miubais doctor lists what this machine has.

uv run miubais start m7-capstone

This copies course/m7-capstone/starter/ to workspace/m7-capstone/ at the repo root. That copy is yours: the stubs under workspace/m7-capstone/src/capstone/ raise NotImplementedError until you fill them in, and the judgment stage asks for a file in the workspace root, named in its description below. The tests stay in course/m7-capstone/tests/ and import your package from the workspace. start --force replaces the workspace with a fresh copy.

uv run miubais check m7-capstone

Runs the stages in order and stops at the first failure, printing its pytest output, so stage 02 waits for stage 01. --stage <stage id> runs one stage and --all keeps going past a failure. A stage whose requires (in stages.yaml) are missing on your machine is skipped, not failed. Results are kept in ~/.miubais/progress.json; uv run miubais reset m7-capstone forgets them (--all for every exercise), and start --force does the same along with the fresh copy.

uv run miubais hint m7-capstone

Takes the last stage check saw fail, its test output and your diff against the starter, and asks the tutor on your Claude Code login for a nudge of under 90 words that is checked not to hand you the solution. If the model cannot be reached, you get a pointer to the failing test instead.

The Run check and Hint buttons on the site’s exercise page do the same while uv run miubais serve is running, and share the progress file with the terminal.

Stages

1. 01-rule-loader: rules.load(repo) -> list[Rule], rules.matches(pattern, path) -> bool and rules.for_files(rules, files) -> list[Rule]. The tests enforce:
   • Loading. Every *.md under <repo>/.claude/rules, recursively, sorted by path relative to that folder. A rule’s name is that relative path without .md (proto, nested/b). A file that starts with a --- line has YAML frontmatter up to the next --- line; it is stripped, and body is the rest with surrounding whitespace removed. Only paths is read; other keys are ignored. paths is a YAML list or a comma-separated string (commas inside braces do not split: "src/**/*.ts, lib/**/*.{ts,tsx}" is two patterns). No paths means paths == []: the rule applies always. A repository without the folder has no rules.
   • Globs. Paths are relative to the repository root with / separators and the whole path must match. * matches within one path segment, ? one character other than /, **/ zero or more whole directories, a trailing ** anything. Braces expand first, nested ones too: web/**/*.{ts,tsx} is web/**/*.ts or web/**/*.tsx. So **/*.ts matches web/src/api.ts and api.ts, src/**/* matches src/c.go but not web/x, and *.json matches CHANGELOG.json but not web/tsconfig.json.
   • Selection. for_files returns the unconditional rules first, then the path-scoped rules with any pattern matching any of the files, each group in load order, each rule once.
2. 02-review-runner: review.FINDINGS_SCHEMA, review.changed_files(patch_text), review.prompt(pr, rules, patch_text), review.ReviewError, review.Reviewer (a protocol: review(pr, tree, rules) -> ReviewResult), review.FakeReviewer(answers_path) and review.AgentReviewer(model=None, query_fn=None, max_turns=20, max_budget_usd=0.40) with options(tree) -> ClaudeAgentOptions, and review.read_guard(tree). The tests enforce:
   • Schema. FINDINGS_SCHEMA is exactly: an object with additionalProperties: false, required: ["findings", "summary"], summary a string of at least 20 characters, and findings an array of objects with additionalProperties: false, required: ["file", "line", "severity", "category", "message", "confidence"], file a string, line an integer ≥ 1, severity one of blocker, major, minor, category one of the twelve CATEGORIES in order, message a string of at least 10 characters, confidence a number in [0, 1] and an optional rule string. Finding enforces the same rules, so a finding outside the enums never enters the pipeline.
   • Changed files. From the diff --git a/<old> b/<new> lines, the +++ b/ headers, --- a/ for deleted files and rename from/rename to lines, in patch order, each once. A pure rename has no ---/+++ headers, and both of its paths count as changed.
   • Prompt. The PR’s id, title and description; the bodies of rules.for_files(rules, changed_files(patch)) and no other rule; the patch itself; and the instruction that only issues the patch introduces count and that file and line refer to the new version of the file, after the patch.
   • Read-only agent. AgentReviewer.review(pr, tree, rules) uses the rules it is given (stage 08 loads them once from the base, never from the patched tree, so a PR cannot write the rules it is reviewed against), builds the prompt from pr.dir / "patch.diff" and runs claude_agent_sdk.query (or query_fn) with cwd=tree, allowed_tools=[] (reads inside cwd need no approval; a bare Read entry would approve reads anywhere on the host), disallowed_tools=["Edit", "Write", "Bash", "WebFetch", "WebSearch", "Agent"], permission_mode="dontAsk", hooks={"PreToolUse": [HookMatcher(matcher="Read|Grep|Glob", hooks=[read_guard(tree)])]}, setting_sources=[], strict_mcp_config=True, output_format={"type": "json_schema", "schema": FINDINGS_SCHEMA}, its max_turns, max_budget_usd and model (claude-opus-5-5 by default). tree is the patched copy that stage 04 prepares; the tests pass the base tree.
   • Read guard. read_guard(tree) is a PreToolUse hook callback: for Read (file_path), Grep (path) and Glob (path and pattern) it resolves each value against tree, expanding ~, and returns a deny (harnesskit.hooks.deny) when it lands outside tree, so /Users/x/.ssh/id_rsa, ~/.aws/credentials and ../ paths are refused; anything else, and every other tool, is allowed ({}).
   • Result. A success result’s structured_output becomes ReviewResult(findings, summary, cost_usd=total_cost_usd, num_turns, session_id, run), where run holds every message of the stream as a harnesskit.testing.Timed (milliseconds since the run started), ready for dump_run. last_cost_usd holds the last result’s cost, also after a failure.
   • Failures. error_max_structured_output_retries (the SDK could not get output that fits the schema), any other error result, a success without structured_output, a success with is_error, output whose findings break the schema, a stream without a result and any SDK exception (ClaudeSDKError, including the ResultError the SDK raises after an error result) all raise ReviewError naming the PR.
   • Fake. FakeReviewer answers from tests/FAKEREVIEW.yaml ({pr_id: {summary, findings}}) with zero cost and an empty run, and raises ReviewError for a PR it has no answer for.
3. 03-filter-and-dedupe: filter.only_touched(findings, changed), filter.dedupe(findings), filter.threshold(findings, min_confidence=0.6), filter.clean(findings, changed, min_confidence=0.6) and filter.reconcile(findings, summary, verify) -> Reconciled(findings, summary, verify_note). Every function returns a new list in input order. The tests enforce:
   • Touched files only. A finding on a file the patch does not touch is dropped: reviewers report pre-existing issues they were told to ignore, and the labels never name such a file.
   • Dedupe. Two findings are one issue when they have the same file, the same category and lines at most 2 apart (|a - b| <= 2); of each issue only the most confident finding stays. Findings of different categories are never merged, even on the same line: a hard-coded token that is also badly named is two findings.
   • Threshold. Findings with confidence >= min_confidence stay.
   • Clean. clean is only_touched, then threshold, then dedupe. On the recorded reviews it keeps 16 of 26 findings.
   • Reconcile. When verify failed (go_ok or ts_ok is False) and the summary claims the tests pass (a sentence about tests, the suite, checks, CI or the build that says pass, green or succeed, with no not, never, fail… or n't between the subject and that word, and no no right before the subject, as in “No tests pass.”; a no elsewhere in the sentence, as in “No regressions, tests pass.”, does not cancel the claim), verify_note says the verify gate failed, names the failing side (go test, node --test) and that the gate overrides the summary. Otherwise it is None. Findings and summary pass through unchanged; the report prints the note next to the summary.
4. 04-verify-gate (needs go and node): verify.apply_patch(repo, patch, dest) -> Applied, verify.run(tree, *, timeout=300) -> VerifyResult and verify.gate(repo, pr, dest=None, *, timeout=300) -> (Applied, VerifyResult | None). The tests run the real toolchain on fixture PRs that pass, fail one side or do not apply, and enforce:
   • Apply. apply_patch copies repo to dest without its top-level prs/ folder (a nested internal/prs/ and .claude/ come along), then runs git apply --check and git apply with the patch inside the copy. When either refuses, ok is False and conflicts lists the files git names in its errors (error: patch failed: <file>:<line>, error: <file>: patch does not apply), each once, in order; the copy stays as it was. tree is dest in both cases.
   • Run. run executes go test ./... and node --test from the tree’s root (Node finds web/**/*.test.ts itself) with timeout seconds each. go_ok/ts_ok is True on exit code 0, False on any other exit code or on a timeout, and None when the tool is not on PATH (a missing tool is not a failure). output holds both commands’ output; duration_ms is the wall time of both.
   • Gate. gate applies pr.patch and runs only when the patch applied: a conflict returns (Applied(ok=False, ...), None) and never runs the tests on the unpatched copy. Without dest it works in a temporary directory and removes it before returning; with dest the patched tree stays there for the reviewer.
5. 05-blocker-ranking: blockers.WEIGHTS, blockers.detect(pr, repo, changed, applied, verify, *, as_of) -> list[Blocker], blockers.score(blockers) -> int, blockers.rank(items) -> list[Ranked] and blockers.to_markdown(ranked) -> str. repo is the base tree (before the patch), changed comes from review.changed_files, applied and verify from stage 04. The tests enforce:
   • Kinds and weights. conflict 100 when the patch did not apply (detail names the conflicting files); verify-failed 90 when verify is not None and go_ok or ts_ok is False (detail names go test and/or node --test; None sides and a missing verify are not failures); stale-migration 80 when a migration the patch adds (migrations/NNNN_*.sql that does not exist in repo) reuses an existing number or is not the next one (the first new migration must be max existing + 1, the next one + 2, and so on; editing an existing migration is not this blocker); missing-second-approval 60 when the patch touches anything under proto/, migrations/ or .claude/ (a PR that edits the review rules or settings changes the gate) and pr.approvals has fewer than two names (a PR that touches none of them is never held for approvals, however few it has); unreviewed-age 30 when pr.approvals is empty and pr.created_at is more than 14 days before as_of (14 days is not enough). Each kind at most once per PR, each Blocker.weight from WEIGHTS. On the fixture with as_of 2026-10-17 every PR gets exactly the blockers in LABELS.yaml.
   • Ranking. score is the sum of the weights. rank takes (pr, blockers) pairs and returns one Ranked(pr_id, score, kinds) per PR, including PRs with no blockers, sorted by score descending and then by id; kinds lists the PR’s blocker kinds heaviest first.
   • Markdown. to_markdown is a table with the header | pr | score | blockers |, a separator row and one row per PR in rank order; the blockers cell is the kinds joined with , , or none.
6. 06-review-metrics: metrics.Label (the shared types.Label, built from a LABELS.yaml entry), metrics.match(findings, labels) -> Matches, metrics.precision_recall(matches, severity=None) -> Score, metrics.blocker_recall(matches) -> float, metrics.summarise(per_pr) -> MetricsReport and metrics.trusted(report) -> bool. The tests enforce:
   • Matching rule. A finding matches a label when file and category are equal and line lies in [start - 3, end + 3]; severity plays no part. Findings are taken in order of confidence, highest first, and each takes the first label (in label order) it matches that is still free: one label, one finding, and one finding, one label. pairs follow label order; unmatched_findings and unmatched_labels keep input order.
   • Scores. Precision is matched findings over findings, recall matched labels over labels, f1 their harmonic mean (0 when both are 0). With severity, precision counts findings of that severity and recall labels of that severity. No findings means precision 1.0, no labels recall 1.0. blocker_recall is the recall over blocker labels.
   • Summary. summarise pools the per-PR matches (a finding only ever matches a label of its own PR) into overall, by_severity for blocker, major and minor, and blocker_recall. trusted is overall precision ≥ 0.6 and blocker recall ≥ 0.9. The recorded reviewer after filter.clean scores precision 11/16, recall 11/12 and blocker recall 1.0, so it is trusted; unfiltered its precision is 12/26 and it is not.
7. 07-review-policy (judgment stage; needs go and node): report.verdict(findings, blockers, reviewed=True), report.run_all (stage 08 describes it) and the CLI python -m capstone.report --repo <path> --out <dir> [--fake | --live] [--as-of 2026-10-17] [--answers tests/FAKEREVIEW.yaml] [--labels tests/LABELS.yaml] [--key tests/KEY.yaml], which prints one line per PR (pr-NN <verdict> verify: … blockers: … reason: …) followed by its cleaned findings and any verify-gate note, and writes REPORT.md under --out. --fake (the default) answers from FAKEREVIEW.yaml, by default course/m7-capstone/tests/FAKEREVIEW.yaml from the repository root. Run it over fixtures/go-ts-mini, read the patches, then write REVIEW_POLICY.md in your workspace root. The tests enforce:
   • Table. A Markdown table | pr | decision | reason | with one row for each of pr-01 … pr-12; decision is merge, hold or block; every row has a reason.
   • Agreement. At least 10 of the 12 decisions agree with tests/KEY.yaml, which is the decision column of LABELS.yaml (a test keeps the two equal).
   • Evidence. The reason of every block names a blocker kind or a category (secret-in-code, conflict, …) that this PR actually has, in its labelled findings or its blockers.
   • Auto-block categories. A section headed ## Auto-block categories lists, one category per bullet (the first word of the bullet, backticks allowed), the categories that block a PR without waiting for a human. It must include secret-in-code and migration-number-collision and must not include style.
   • CLI. python -m capstone.report --fake exits 0 and prints a verdict line for each of the twelve PRs.
8. 08-live-review (live; needs a logged-in Claude Code, go and node): report.run_all(repo, prs, reviewer, *, as_of, out_dir, run_tests=True, labels=None, key=None) -> Report with report.PRReport and report.Report as given in the starter. The offline tests run it with the recorded reviewer and enforce:
   • Per PR. run_all loads the rules once from repo, the base, and passes them to every review. With run_tests=True the patch is applied and tested with verify.gate into a temporary directory and the reviewer reads that patched tree; with run_tests=False the patch is only applied. A patch that does not apply is not reviewed (review == "skipped"). Findings go through filter.clean, the summary through filter.reconcile, blockers come from blockers.detect and are listed heaviest first. verify is pass, fail, conflict, skipped (no toolchain) or not run.
   • Verdict. block when any finding has severity blocker or the PR has a conflict, stale-migration or verify-failed blocker; otherwise hold when it has a major finding, a missing-second-approval or unreviewed-age blocker, or its review failed (reviewed=False: a failed review is never a merge); otherwise merge. PRReport.reason names what produced the verdict: the heaviest deciding blocker with its detail, else the first deciding finding (block: blocker finding secret-in-code at web/src/api.ts:3), else hold: review failed, else merge: no blocker, no major finding. With the recorded reviewer every verdict equals the label’s decision.
   • Failures. A ReviewError is retried once. A second failure records the PR with review == "review-failed", no findings and the error, and the run goes on. cost_usd adds up every attempt (last_cost_usd of a reviewer that failed) and num_turns comes from the ReviewResult.
   • Outputs. <out_dir>/REPORT.md: a table | pr | verdict | reason | verify | blockers | findings | cost_usd | turns |, a ## Blocker ranking section (report.RANKING_NOTE, which says the ranking covers repository signals only, then blockers.to_markdown), a ## Metrics against the labels section when labels is given, and a ## pr-NN: <title> section per PR with its verdict, Verify: <status>, Reason: <reason>., blockers, findings as `file:line` and the verify-gate note, so a summary that claims the tests pass never outweighs a failed gate. <out_dir>/runs/pr-NN.jsonl is the review’s run through harnesskit.testing.dump_run, after a meta line. <out_dir>/runs/live-<UTC date>.json is the aggregate, in the shape of fixtures/live_review_run.json: the reviewer’s class, model, max_budget_usd_per_pr and max_turns (None for the fake); per PR the verdict, reason, review and verify status, blocker kinds, finding count, cost, turns and time; metrics against the labels; verdicts_agreeing_with_key when key (the decisions of KEY.yaml) is given; totals of cost, turns, wall time, PRs by verdict and failed reviews.
   • Live. The live test runs AgentReviewer (claude-opus-5-5, max_budget_usd=0.40, max_turns=20) over the twelve PRs with your workspace as out_dir and asserts 12 rows, blocker recall ≥ 0.75, overall precision ≥ 0.4 and a total cost of at most 5 USD. The recorded run cost 0.70 USD and took about 2 minutes; the test allows up to 5 USD.

Requirements by stage

Stage	Title	Read first	Live	Requires
01-rule-loader	Path-scoped rules from .claude/rules	Conventions of a Go, TypeScript, proto and migrations repository	no	-
02-review-runner	A read-only review runner with structured findings	Reading a PR with a read-only agent	no	-
03-filter-and-dedupe	Finding hygiene: touched files, dedupe, confidence and the verify gate's word	Review as a measurable pipeline	no	-
04-verify-gate	Apply the patch to a copy and run go test and node --test	Blocker ranking from repository signals	no	go, node
05-blocker-ranking	Blockers from repository signals, ranked per PR	Blocker ranking from repository signals	no	-
06-review-metrics	Precision and recall against labels	Review as a measurable pipeline	no	-
07-review-policy	Review policy for the twelve PRs (judgment stage)	Blocker ranking from repository signals	no	go, node
08-live-review	Live: review all twelve PRs and write the report	From pre-review to what blocks prod	yes	claude-auth, go, node

Check your work

Check your work from the clone:

uv run miubais check m7-capstone
uv run miubais hint m7-capstone

Opened from pnpm dev on localhost with uv run miubais serve running, this panel runs them for you.