A Comparison and Critical Reflection of Information Disorder Detection Techniques: Performing a Cross-Data and Cross-Model Evaluation
Information disorders, such as dis-, mis-, and malinformation, can lead to societal and/or economic harm. They are rapidly spread, extensively consumed on the web, and represent a threat to democracy. AI-based detection models can identify information disorders to some extent. However, major issues are the dynamics of news characteristics and concept drift. The generalization ability of a model is an important requirement and refers to its robustness when applied on unseen data. The aim of this work is to better understand the state of the art regarding information disorder detection approaches by conducting a reproducibility study and a cross-data and cross-model comparative analysis that leads to: (i) insights with respect to the effectiveness of binary information disorder classification; (ii) performance results on seen and unseen data; and (iii) new mixed European datasets named MENA.
We conduct an evaluation of a fine-tuned BERT-based model applied on European data, which has received limited attention to date. The best performing models in our experiments are the RoBERTa and the Longformer models. The evaluation gives insights about potential biases of datasets that can be used to improve a model’s generalization ability. We also show that using domain-specific datasets for fine-tuning contributes to the robustness of models. Finally, we provide takeaways concerning reproducibility and stress the need for more transparent AI-based detection techniques.
Biography
Dr. Sabrina Kirrane is an associate professor at the Vienna University
of Economics and Business, where she is a member of the Institute for
Complex Networks, the Competence Center for Applied AI and Scientific
Computing, and the Research Institute for Crypto Economics. Her research
interests include legal information extraction and knowledge
representation; machine actionable policies, automated compliance
checking, and governance; information disorder detection; algorithmic
transparency, explainability, and bias detection; and intelligent
software agents. Dr. Kirrane co-founded the Society, Privacy and the
Semantic Web - Policy and Technology (PrivOn) workshop series, which was
co-located with the International Semantic Web Conference between 2013
and 2017. Additionally, she is on the editorial board of the Semantic
Web Journal and the Transactions on Graph Data and Knowledge. Over the
years, she has also been on the organising committee of the ISWC, ESWC,
RuleML+RR, and Semantics.
