Call for papers: 1st Workshop on Novel Evaluation Approaches for Text Classification Systems on Social Media

The automatic or semiautomatic analysis of textual data is a key approach to analyse the massive amounts of user-generated content online, from the identification of sentiment in text and topic classification to the detection of abusive language, misinformation or propaganda. However, the development of such systems faces a crucial challenge. Static benchmarking datasets and performance metrics are the primary method for measuring progress in the field, and the publication of research on new systems typically requires demonstrating an improvement over state-of-the-art approaches in this way. Yet, these performance metrics can obscure critical failings in current models. Improvements in metrics often do not reflect improvements in the real-world performance of models. There is clearly a need to rethink performance evaluation for text classification and analysis systems to be usable and trustable.

If unreliable systems achieve astonishing scores with traditional metrics, how do we recognize progress when we see it? The goal of the workshop on Novel Evaluation Approaches for Text Classification Systems on Social Media (NEATCLasS) is to promote the development and use of novel metrics for abuse detection, hate speech recognition, sentiment analysis, and similar tasks within the community, to better be able to measure whether models really improve upon the state of the art, and to encourage a wide range of models to be tested on these new metrics.

We invite research papers (8 pages), position and short papers (4pages), and demo papers (2 pages). Submissions must be original and should not have been published previously or be under consideration for publication while being evaluated for this workshop. Submissions will be evaluated by the program committee based on the quality of the work and its fit to the workshop themes. All submissions should be double-blind and a high-resolution PDF of the paper should be uploaded to the easy chair submission site (link below) before the paper submission deadline. All papers must be submitted, and formatted in AAAItwo-column, camera-ready style. Authors of accepted papers will have the opportunity to publish their papers through workshop proceedings by the AAAI Press.

Submission instructions will be uploaded to the workshop webpage in due course: https://neatclass-workshop.github.io

Timeline

* Submission link: https://easychair.org/conferences/?conf=neatclass2022

* UPDATED Papers submission deadline: April 10, 2022

* UPDATED Paper acceptance notification: April 29, 2022

* UPDATED Final camera-ready paper due: May 6, 2022

* Workshop Day: June 6, 2022

All deadlines are 11:59pm AOE (anywhere on earth).

Organizers

Björn Ross, University of EdinburghRoberto Navigli, Sapienza University of RomeAgostina Calabrese, University of Edinburgh University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannaisa th' ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidhSC005336.