MCFEND: A Multi-source Benchmark Dataset for Chinese Fake News Detection

Yupeng Li1, Haorui He1,2, Jin Bai3, Dacheng Wen1,4
1Hong Kong Baptist University
2The Chinese University of Hong Kong, Shenzhen 3Beijing Normal University-Hong Kong Baptist University United International College 4The University of Hong Kong
Paper arXiv Dataset

MCFEND, the initial multi-source Chinese fake news detection dataset, comprises multi-modal content and social context of 23,974 real-world Chinese news pieces collected from diverse sources such as social platforms, messaging apps, and traditional online news outlets.

Teaser image.

Fake news originating from diverse sources differs in both content and social context.

Textual Feature
Visualization for Textual Feature
Textual Feature
Visualization for Social Emotion

Abstract

The prevalence of fake news across various online sources has had a significant influence on the public. Existing Chinese fake news detection datasets are limited to news sourced solely from Weibo. However, fake news originating from multiple sources exhibits diversity in various aspects, including its content and social context. Methods trained on purely one single news source can hardly be applicable to real-world scenarios. Our pilot experiment demonstrates that the F1 score of the state-of-the-art method that learns from a large Chinese fake news detection dataset, Weibo-21, drops significantly from 0.943 to 0.470 when the test data is changed to multi-source news data, failing to identify more than one-third of the multi-source fake news.

To address this limitation, we constructed the first multi-source benchmark dataset for Chinese fake news detection, termed MCFEND, which is composed of news we collected from diverse sources such as social platforms, messaging apps, and traditional online news outlets. Notably, such news has been fact-checked by 14 authoritative fact-checking agencies worldwide. In addition, various existing Chinese fake news detection methods are thoroughly evaluated on our proposed dataset in cross-source, multi-source, and unseen source ways. MCFEND, as a benchmark dataset, aims to advance Chinese fake news detection approaches in real-world scenarios.

BibTeX

@INPROCEEDINGS{mcfend,
      title={MCFEND: A Multi-source Benchmark Dataset for Chinese Fake News Detection}, 
      author={Li, Yupeng and He, Haorui and Bai, Jin and Wen, Dacheng},
      booktitle={Proc.~of WWW (to apear)}, 
      year={2024},
}