RL002
100% precise
1 of 1 finding judged worth fixing.
Too few findings to say anything about this rule yet.
Evaluation
What it actually gets right.
The metric this project is organised around is Precision@3: of the top 3 findings on a paper, how many would the paper's own authors agree are worth fixing? Three findings someone acts on beat thirty they skim. Recall is not measured, because a rule that stays silent is an acceptable outcome and a noisy rule is not.
100%
precision@3
1/1
top-3 findings judged worth fixing
1papers labelled by hand
1findings judged
Sample: 1 paper, 1 finding, judged by a human against the question "would the authors agree this needs fixing?". This is a small sample. Treat the rates below as indicative, not as estimates with meaningful confidence intervals.
By rule
A rule that is bad is listed as bad. The invalid findings are quoted with the reason they were rejected, because a false positive you can read is more useful than a rate you have to trust.
Shipped but never judged: RL001, RL003, RL004, RL005
These rules ran on the corpus but no human verdict was recorded for any of
their findings, so nothing on this page speaks to their quality. That is a
gap in the evaluation, not a clean bill of health.
Label sources: labels.json
How the measurement is done
The method, so you can discount it properly.
- what is judged
- Each finding in a paper's ranked list, in order, by a human reading the paper. Not by a model, and not by the rule that emitted it.
- the question asked
- "Would this paper's authors agree this is worth fixing?" A finding that is technically true but pointless counts as invalid.
- what is not measured
- Recall. There is no attempt to find every problem in a paper, so a rule that fires on nothing is not penalised — but a rule that fires on the wrong thing is.
- the ordering
- Precision@3 uses the pipeline's own ranking — severity, then confidence, then rule id — the same order the report page and the command line use. "Top 3" means one thing everywhere.
- what this cannot tell you
- How the rules behave on papers outside the corpus. Every paper here is a recent arXiv preprint in machine learning, and a rule tuned on twelve papers is not evidence about the field.