Novel Claim or Déjà Vu? Rethinking "Contamination-Free" Dynamic Evaluation for Multimodal Automated Fact-Checking

Haorui He1,2, Xinwen Chen3, Dacheng Wen1,2, Reynold Cheng2, Francis C. M. Lau2, Yupeng Li1,*
1Department of Interactive Media, Hong Kong Baptist University
2School of Computing and Data Science, The University of Hong Kong
3Faculty of Science and Technology, Beijing Normal-Hong Kong Baptist University
*Correspondence: ivanypli@gmail.com
Code arXiv (appendix)

This work revisits the assumption that dynamic evaluation is inherently contamination-free for multimodal automated fact-checking (MAFC). We empirically study contamination risks in both static and dynamic benchmarks and reveal their impact on MAFC evaluation.

Rethink-MAFC-Eval Dynamic Evaluation Framework Overview.

Abstract

Multimodal automated fact-checking (MAFC) verifies claims by retrieving and reasoning over external evidence. However, most existing static benchmarks risk contamination: they primarily consist of outdated claims verifiable using an LLM's internal knowledge without external evidence. This can inflate performance estimates and fail to reflect true capability on novel claims that require up-to-date information. To address this, emerging dynamic benchmarks collect claims published after LLMs' knowledge cut-off dates, assuming they are uncontaminated. This work revisits this assumption by empirically studying contamination risks in both the state-of-the-art (SOTA) static AVeriTeC benchmark and our newly constructed dynamic ClaimReview2025Q4 benchmark, as well as their impact on MAFC evaluation.

Our experiments yield 16 findings, highlighting three key results: (1) Dynamic evaluation reduces but does not eliminate contamination risks, as 17.09%-29.30% of post-cut-off claims remain potentially contaminated; (2) Many newly published claims can be verified either directly or by synthesizing multiple pieces of public knowledge available before the cut-off; and (3) Contamination can induce statistically significant inflation in MAFC performance, increasing Macro-F1 by up to 11.34 points and distorting system rankings. In light of these findings, we re-evaluate SOTA LLMs under a strictly contamination-controlled setting. Our study provides practical guidelines for trustworthy MAFC evaluation.

Contamination Detection Pipeline

An overview of contamination detection via an evidence sufficiency evaluation pipeline.

Empirical Findings

We summarize our experimental findings according to RQ1-RQ4.

RQ1: To what extent are existing static and dynamic MAFC benchmarks contaminated?

  • Finding 1.1:The dynamic benchmark can effectively alleviate data contamination.
  • Finding 1.2:Strong contamination signals are widespread across LLMs on AVeriTeC; on ClaimReview2025Q4, peak scores are concentrated in particular models.
  • Finding 1.3:A non-trivial portion of claims in the dynamic benchmark still faces contamination risk across all models.
  • Finding 1.4:Semantic metrics identify a markedly larger contaminated subset than lexical matching.
  • Finding 1.5:Qwen3.5-122B-A10B is the most contaminated model on the dynamic benchmark.
  • Finding 1.6:A subset of claims in the dynamic benchmark exhibits extremely high contamination risk.

RQ2: How does contamination arise in dynamic benchmarks?

  • Finding 2.1:Contamination arises from claims that directly reference facts available before the LLMs' knowledge cut-off dates.
  • Finding 2.2:Contamination can also result from synthesizing multiple pre-cut-off facts.

RQ3: How does contamination affect MAFC performance evaluation?

  • Finding 3.1:Contamination can significantly inflate evaluation results.
  • Finding 3.2:The magnitude of this inflation varies substantially across models.
  • Finding 3.3:The most contaminated model also suffers the largest Accuracy decline.
  • Finding 3.4:Contamination can distort MAFC performance rankings of LLMs.
  • Finding 3.5:Contamination enables models to retrieve relevant evidence more precisely, bypassing comprehensive exploration of the evidence space.

RQ4: How do SOTA LLMs perform under contamination-controlled MAFC evaluation?

  • Finding 4.1:The evaluation set exhibits noticeable label imbalance.
  • Finding 4.2:DeepSeek-V3.2 achieves the strongest overall MAFC performance in the contamination-controlled setting.
  • Finding 4.3:Significant room for improvement remains in MAFC.

BibTeX

@inproceedings{rethink_mafc_eval_2026,
  title={Novel Claim or Déjà Vu? Rethinking "Contamination-Free" Dynamic Evaluation for Multimodal Automated Fact-Checking}, 
  author={He, Haorui and Chen, Xinwen and Wen, Dacheng and Cheng, Reynold and Lau, Francis C. M. and Li, Yupeng},
  booktitle={Proc.~of MM},
  year={2026},
}