{"paper":{"id":"2609.00067.pdf","title":"Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy","authors":["Yi-Cheng Lai","Hen-Hsen Huang"],"arxiv_id":"","url":"","source_kind":"pdf","ir_version":"1.0"},"generated_at":"2026-09-14T04:22:17.595598+00:00","counts":{"error":0,"warning":0,"info":0},"findings":[],"top_findings":[],"ir_summary":{"claims":25,"contributions":4,"method_components":11,"experiments":12,"baselines":4,"datasets":5,"metrics":4,"tables":13,"figures":5,"citations":24,"equations":0,"quantitative_results":517,"has_repository":false},"ir_health":{"observations":[{"code":"numbers-are-identifier-fragments","severity":"note","message":"12 extracted numbers are digits inside identifiers.","detail":"e.g. '2.5' in 'GPT-5.1 Gemini 2.5 Qwen3-Thinking Claude', '2.5' in 'GPT-5.1 Gemini 2.5 Qwen3-Instruct Kimi-K', '2.5' in 'GPT-5.1 Gemini 2.5 Qwen3-Thinking Claude'. Rules that compare numbers across sections can report a contradiction between an identifier and a real value."},{"code":"compile-warning","severity":"note","message":"references: the reference list carries no entry numbers; citation keys are enumeration order [1]..[N], not labels printed in the paper, so a key may not match the [N] used in the body","detail":""}],"errors":0,"warnings":0,"trustworthy":true},"diagnostics":{"source":"/home/oceann/workspace/research-linter/eval/papers/2609.00067.pdf","rules_run":["RL001","RL002","RL003","RL004"],"rules_skipped":[{"rule":"RL005","reason":"not-applicable"}],"rule_errors":{},"rule_findings":{"RL001":0,"RL002":0,"RL003":0,"RL004":0},"rule_seconds":{"RL001":0.02,"RL002":0.84,"RL003":1.28,"RL004":0.02},"dropped_no_evidence":0,"demoted_low_confidence":0,"ir_health":{"observations":[{"code":"numbers-are-identifier-fragments","severity":"note","message":"12 extracted numbers are digits inside identifiers.","detail":"e.g. '2.5' in 'GPT-5.1 Gemini 2.5 Qwen3-Thinking Claude', '2.5' in 'GPT-5.1 Gemini 2.5 Qwen3-Instruct Kimi-K', '2.5' in 'GPT-5.1 Gemini 2.5 Qwen3-Thinking Claude'. Rules that compare numbers across sections can report a contradiction between an identifier and a real value."},{"code":"compile-warning","severity":"note","message":"references: the reference list carries no entry numbers; citation keys are enumeration order [1]..[N], not labels printed in the paper, so a key may not match the [N] used in the body","detail":""}],"errors":0,"warnings":0,"trustworthy":true},"notes":["RL001 funnel arithmetic: sentences_with_from=7 endpoint_pairs=2 claims_compared=0 emitted=0 [dropped: not-a-measurement=0]","RL001 funnel delta-column: structured_tables=11 with_exactly_one_delta_column=0 rows_checked=0 emitted=0","RL001 funnel cross-location: quant=517 prose=263 prose_with_metric=50 prose_with_dataset=3 prose_with_metric_and_dataset=0 distinct_buckets=0 emitted=0 [dropped: not-a-measurement=3 not-in-quote=8]","RL001 funnel identifier-drift: settings_checked=2 sections_stating_one_value=0 emitted=0","RL002 citation entailment: attempted 14 of 14 ranked cited claims (47 in-text markers from 24 bibliography entries); 8 cited works could not be identified, 3 had no retrievable text, 0 model quotes failed verbatim verification, 0 were topically unrelated to the retrieved text.","RL002 candidate ranking (why the top 14 were chosen): [11] [introduction, no DOI/arXiv id]; [22] [introduction, no DOI/arXiv id]; [10] [introduction]; [21] [introduction]; [15] [discussion, names an artefact (The GPT-4o), unusable title (identity fragile)]; [15] [experiment, absolutist wording, title/sentence overlap near zero, unusable title (identity fragile)]; [3] [results, title/sentence overlap near zero, unusable title (identity fragile)]; [15] [discussion, unusable title (identity fragile)]","RL002 label distribution over 3 classified citations: SUPPORTED=3","RL002 base-rate self-check: not applicable (3 classified, needs >=5).","RL002 attribution check (case B): 9 candidates name an artefact, 4 of them made the top-N and were checked against their cited source","RL002: 4 markers skipped because the claim is jointly attributed to several works","RL002: 3 enumerated-citation sentences skipped (they cite >=3 works, so the claim is about the set and cannot be refuted by any single cited work)","RL002: 5 duplicate candidates dropped (same reference and claim, or the same artefact attributed twice)","RL002: 8 cited works unresolvable (first 3: [{\"key\": \"[15]\", \"title\": \"GPT-4o system card\", \"why\": \"bibliography title unusable for identity verification\"}, {\"key\": \"[15]\", \"title\": \"GPT-4o system card\", \"why\": \"bibliography title unusable for identity verification\"}, {\"key\": \"[3]\", \"title\": \"Qwen3-vl technical report\", \"why\": \"bibliography title unusable for identity verification\"}])","RL003: 3 queries -> 179 unique works (arXiv search 104, category listings 0, scholarly graph 75 [crossref 75], 0 graph works matched to an arXiv id); source errors during retrieval: arXiv 0, graph 0","RL003: parsed the reference list of 11/12 comparable papers (1 unreadable) — that is the PeerUsage denominator","RL003: enriched 1 of 1 candidates with their own abstract (0 unavailable, 0 over budget); 0 dropped as published after this paper","RL003: scholarly graph for this run = crossref","RL003 funnel (636 candidates): already_used_by_the_paper -3, cited_only_from_related_work -128, is_a_dataset -5, is_a_metric -1, peer_usage_below_quorum -159, survey_or_artifact -45, title_names_no_method -294 || kept: citation_context_unknown +64 || survived-cheap-gates 1 -> with-abstract 1 -> comparable 1 -> above-threshold-0.65 0","RL003 candidate: 'Decoupled weight decay regularization' score 0.246 (T 0.00 D 0.50 M 0.33 P 0.27 R 0.00; 3/11 peers cite it) -> not reported","RL003: 1 comparable candidates all scored below 0.65 — nothing to report for this paper","RL004 funnel: components=11 contribution_backed=2 candidates=0 ablations_found=3 matched_by_ablation=0 unmatched=0 experiments=12","RL004 component M1 '998-case context-conditioned diagnostic benchmark': covered (filtered earlier)","RL004 component M2 'System-2 Visual Arbitration (S2VA)': covered (filtered earlier)","RL004 component M3 'Context-blind witness': covered (filtered earlier)","RL004 component M4 'System 2 Visual Arbiter': covered (filtered earlier)","RL004 component M5 'Leaky Witness': covered (filtered earlier)","RL004 component M6 'Witness-Only': covered (filtered earlier)","RL004 component M7 'Visual Supremacy Only': covered (filtered earlier)","RL004 component M8 'Evidence separation': covered (filtered earlier)","RL004 component M9 'CoVe-style verification proxy': covered (filtered earlier)","RL004 component M10 'Two-Call Describe–Answer': covered (filtered earlier)","RL004 component M11 'Single-Call Describe–Answer': covered (filtered earlier)","RL004 funnel: ranked=0 emitted=0 suppressed_by_cap=0 dropped_unverified_quote=0"],"missing_core_rules":[],"seconds":18.65,"llm":{"calls":0,"cache_hits":4,"errors":0,"prompt_chars":0},"sources":{"arxiv":{"requests":0,"errors":0,"cache_hits":41,"available":1},"openalex":{"requests":1,"errors":0,"cache_hits":0,"available":0,"unavailable_reason":"OpenAlex free daily budget exhausted (resets midnight UTC)"},"crossref":{"requests":0,"errors":0,"cache_hits":12,"available":1},"s2":{"requests":0,"errors":0,"cache_hits":0,"available":1},"short_circuited":["crossref","openalex","s2"]},"cache":{"hits":57,"misses":1}},"compile_warnings":["references: the reference list carries no entry numbers; citation keys are enumeration order [1]..[N], not labels printed in the paper, so a key may not match the [N] used in the body"]}