Rule catalogue

What it checks, and what it refuses to check.

Every rule is one file, one class, and one question about the compiled paper. A rule that is unsure emits nothing — a conservative rule that finds nothing is an acceptable outcome, a noisy rule is not.

RL001

Result consistency

deterministic runs offline

Do the paper's numbers agree with each other?

Detects

The same quantity reported with different values across the abstract, body, tables and appendix.
Arithmetic that does not follow: the text claims +8.3% but the paper's own table shows 74.2 to 80.1.
An identifier — dataset, metric, seed, sample size — named one way in the text and another in a table or caption.
Cross-references that dangle: a table referenced before it exists, a figure never referenced at all.

How it decides

Values are harvested from prose and table cells with their location, bucketed by metric and dataset, then compared in Python. Deltas are recomputed in code, never by a model.

RL002

Citation entailment

uses a model for meaning needs network

Does the cited work actually say what the claim says?

Detects

Scope overgeneralisation: a claim that a cited paper showed a general result when it showed one on two synthetic benchmarks.
Claims attached to a citation the cited abstract does not support.
Claims the cited work directly contradicts.

How it decides

The cited work is resolved through OpenAlex and arXiv, then the passages with the highest overlap with the claim are retrieved — never the whole paper. A model classifies the relation; its quote must be a verified substring of the passages supplied or the finding is dropped.

RL003

Missing baseline

uses a model for meaning needs network

Is there a comparable method the paper should have run against?

Detects

A published method that evaluates the same task on the same dataset with the same metric and is missing from the comparison.
Above all when the paper's own related-work cluster benchmarks against it.

How it decides

Candidate baselines come from the scholarly graph, filtered by measured task, dataset and metric overlap with this paper's experiments. The overlap scores are shown so the claim can be checked rather than believed.

RL004

Missing ablation

deterministic runs offline

Is every claimed contribution actually isolated?

Detects

A component presented as a contribution that no reported ablation independently removes or replaces.

How it decides

Claimed contributions are matched against reported ablations by component identity. Absence is only reported alongside the search that established it, and never when extraction warnings say the ablation table failed to parse.

RL005

Paper / code mismatch

deterministic needs network

Does the released code do what the paper says it does?

Detects

A hyperparameter the paper states explicitly — batch size, learning rate, optimiser, epochs, seed — that the configuration in the paper's own linked repository disagrees with.
A configuration the repository leaves ambiguous across several files, reported as a question rather than as a mismatch.

How it decides

Only a fixed whitelist of hyperparameters is compared, each with one unambiguous meaning. The paper's value must be an explicit number in an experimental section near a first-person framing; the repository's value must be single-valued. Gradient accumulation is honoured, so the figure compared is the effective batch size. Every finding shows both sides, and says plainly that a released config is often the documented default the authors' run overrode.

The admissibility gate

Nothing reaches you without evidence.

Between a rule and your screen sits one gate, applied on every run. It is the reason a finding here is worth reading.

evidence required: A finding whose evidence all has an empty quote and an empty URL is dropped before rendering. Those are the only two fields the gate reads: for computed evidence the quote holds the arithmetic ("80.1 − 74.2 = 5.9, but the abstract says +8.3%"), and the numbers alongside it are supplementary. Dropped findings are counted, and the count is printed on the report.
confidence demotion: A finding claimed as an error that does not clear 0.7 confidence is demoted to a warning automatically, on every run, and the number demoted is printed on the report.
the rule-author convention: Rules are written so that anything they cannot justify above 0.5 is dropped or marked INFO by the rule itself. That one is a convention, not a gate: nothing in the pipeline checks it, because a threshold cannot tell a well-calibrated 0.45 from a badly-calibrated one. It is a property of how these rules were written rather than a guarantee about every finding, and a rule that ignored it would pass the gate unhindered.
quotes are verbatim: A model-produced quote is only used if it is an exact substring of the text the model was given. Paraphrase is treated as fabrication.
one ranking everywhere: Findings are ordered by severity, then confidence, then rule id. The terminal output, the stored JSON, this report page and the evaluation harness all use that same order, so “the top three” means one thing.

Credits

Photography and type.

Every photograph on this site is a real image from a public archive or stock library, used under the licence shown. There are no drawn marks, no icon fonts and no generated graphics anywhere in this interface.

Photography

catalog-cards: Card catalog at the Indiana State Library — interior view of catalog cards. TBurmeister (WMF), Wikimedia Commons, CC BY-SA 4.0.
catalog-cabinet: Card catalog at the Indiana State Library — cabinet exterior. TBurmeister (WMF), Wikimedia Commons, CC BY-SA 4.0.
punch-cards: Punched card program deck. Arnold Reinhold, Wikimedia Commons, CC BY-SA 3.0.
type-case: Letterpress type case. Wikimedia Commons, CC BY-SA 4.0.
stacks-dark: Library stacks, dark. Pexels, free to use under the Pexels licence.
site icon: A crop of the punched-card photograph above, resized to 32, 180, 192 and 512 px. Derived from a real photograph rather than drawn, and carrying the same CC BY-SA 3.0 terms.

Type

Newsreader: Display serif — headlines, section titles, evidence quotes. Production Type, SIL Open Font License.
Archivo: Text grotesque — body copy, navigation, buttons, labels. Omnibus-Type, SIL Open Font License.
Martian Mono: Monospace — rule ids, stat figures, locations, log output. Evil Martians, SIL Open Font License.
arrows: One glyph: the transformation arrow in paper.pdf → PaperIR. None of the three families carries an arrow, so without this face the browser draws it with whatever monospace the visitor's operating system supplies — differently on every machine. Noto, SIL Open Font License.
self-hosted: All four are served as woff2 subsets from this origin, built by web/tools/fetch_fonts.py — variable axes this design never moves are pinned, and the glyph set is cut to what the templates can actually print. Loading this page fetches about 100 KB of type, and nothing is requested from a font CDN.