CLEF 2021 Conference and Labs on the Evaluation Forum
Information Access Evaluation meets Multilinguality, Multimodality and Visualization
21 - 24 September 2021, Bucharest - Romania
*** CALL FOR LAB PARTICIPATION ***
CLEF 2021 is the 12th CLEF conference continuing the popular CLEF campaigns which have run since 2000 contributing to the systematic evaluation of information access systems. It consists of a peer-reviewed conference (see the separate call for papers) and a set of 12 Labs designed to test different aspects of multilingual and multimedia IR systems:
1. ARQMath: Answer Retrieval for Questions on Math
2. BioASQ: Large-scale biomedical semantic indexing and question answering
3. CheckThat!: Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News
4. ChEMU: Cheminformatics Elsevier Melbourne University lab
5. CLEF eHealth: medical content retrieval to support decision making
6. eRisk: Early Risk prediction on the Internet
7. ImageCLEF: Multimedia Retrieval Challenge in CLEF
8. LifeCLEF: Multimedia Life Species Identification
9. LiLAS: Living Labs for Academic Search
10. PAN: Digital Text Forensics and Stylometry
11. SimpleText-202: (Re)Telling right scientific stories to non-specialists via text simplification
12. Touché: Argument Retrieval
*** IMPORTANT DATES ***
Lab Registration Opens: 16 November 2020
Registration closes: 30 April 2021
End Evaluation Cycle: 7 May 2021
Working Notes papers due: 28 May 2021
Camera Ready Copy: 2 July 2021
*** PARTICIPATION ***
*** LABS ***
1. ARQMath: Answer Retrieval for Questions on Math
ARQMath aims to advance math-aware search and the semantic analysis of mathematical notation and texts.
Task 1 - Answer Retrieval: Given a math question post, return relevant answer posts.
Task 2 - Formula Search: Given a formula in a math question post, return relevant formulas from both question and answer posts.
Lab Coordination: Richard Zanibbi (Rochester Institute of Technology), Douglas Oard (University of Maryland), Anurag Agarwal (Rochester Institute of Technology), and Behrooz Mansouri (Rochester Institute of Technology)
Twitter: @ARQMath1
2. BioASQ: Large-scale biomedical semantic indexing and question answering
The aim of the BioASQ Lab is to push the research frontier towards systems that use the diverse and voluminous information available online to respond directly to the information needs of biomedical scientists.
Task 1 - Task A: Large-Scale Online Biomedical Semantic Indexing.
Task 2 - Task B: Biomedical Semantic Question Answering.
Task 3 - Task MESINESP: Medical Semantic Indexing In Spanish.
Task 4 - Task Synergy: Question Answering for COVID-19.
Lab Coordination: Anastasia Krithara (National Center for Scientific Research "Demokritos"), Anastasios Nentidis (National Center for Scientific Research "Demokritos" & Aristotle University of Thessaloniki), George Paliouras (National Center for Scientific Research "Demokritos"), Martin Krallinger (Barcelona Supercomputing Center), and Marta Villegas (Barcelona Supercomputing Center)
Twitter: @BioASQ
3. CheckThat!: Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News
The CheckThat! Lab aims at fighting misinformation and disinformation in social media, in political debates, and in the news, with focus on three tasks (in English, Arabic, German and Spanish): check-worthiness estimation, detecting previously fact-checked claims, and fake news detection.
Task 1 - Check-worthiness estimation: given a piece of text, detect whether it is worth fact-checking.
Task 2 - Detecting previously fact-checked claims: given a check-worthy claim in the form of a tweet, and a set of previously-checked claims, determine whether the claim has been previously fact-checked.
Task 3 - Fake new detection: this task has two subtasks: multi-class fake news detection, and domain identification of fake news articles.
Lab Coordination: Preslav Nakov, Giovanni da San Martino, Tamer Elsayed, Alberto Barron-Cedeno,Ruben Miguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Nikolay Babulkov, Alex Nikolov, Thomas Mandl, Julia Maria Struß, Gautam Kishore Shahi, and Sandip Modha
4. ChEMU: Cheminformatics Elsevier Melbourne University lab
ChEMU (Cheminformatics Elsevier Melbourne University) lab aims to advance the state-of-the-arts in information extraction over chemical patents, and provide the opportunity to address the fundamental NLP task - anaphora resolution in the context of chemical patents.
Task 1 - Chemical Reaction Reference Resolution: given a description of a chemical reaction, the task requires identification of other chemical reactions or general conditions that it refers to.
Task 2 - Anaphora Resolution: this task focuses on expression-level references, and requires identification of reference resolution between expressions in chemical patents.
Lab Coordination: Karin Verspoor (The University of Melbourne), Tim Baldwin (The University of Melbourne), Trevor Cohn (The University of Melbourne), Saber Akhondi (Elsevier), Jiayuan He (The University of Melbourne), Christian Druckenbrodt (Elsevier), Camilo Thorne (Elsevier), Biaoyan Fang (The University of Melbourne), Hiyori Yoshikawa (The University of Melbourne), and Zenan Zhai (The University of Melbourne)
5. CLEF eHealth: medical content retrieval to support decision making
This lab aims to support the development of techniques to aid laypeople, clinicians and policy-makers in easily retrieving and making sense of medical content to support their decision making.
Task 1 - Information Extraction from Noisy Text: participants will identify and classify Named Entities in written ultrasonography reports, containing misspellings and inconsistencies, from a major public hospital in Argentina.
Task 2 - Consumer Health Search: participants must retrieve web pages that fulfil a given patient’s personalised information need. This needs to fulfil the following criteria: information credibility, quality, and suitability.
Lab Coordination: Laura Alonso Alemany (Univ. Nacional de Córdoba), Viviana Cotik (Univ. de Buenos Aires), Lorraine Goeuriot (Univ. J.Fourier), Liadh Kelly (Maynooth University), Gabriella Pasi (University of Milano-Bicocca), and Hanna Suominen (Australian National Univ.)
Twitter: @clefhealth
6. eRisk: Early Risk prediction on the Internet
The main purpose of eRisk is to explore issues of evaluation methodology, effectiveness metrics and other processes related to early risk detection.
Task 1 - Early detection of pathological gambling: this task will pursue early detection models for emitting alerts on users starting to develop this mental health problem.
Task 2 - Early detection of self-harm: this task will pursue early detection models for emitting alerts on users starting to develop this mental health problem.
Task 3 - Depression-level estimation: in this task, participants are expected to develop models to automatically fill a standard depression questionnaire based on the user's writings.
Lab Coordination: Javier Parapar (Universidade da Coruña), Patricia Martín-Rodilla (Universidade da Coruña), David E. Losada (Universidade de Santiago de Compostela), and Fabio Crestani (University of Lugano)
Twitter: @earlyrisk
7. ImageCLEF: Multimedia Retrieval Challenge in CLEF
ImageCLEF 2021, the Multimedia Retrieval Challenge in CLEF, is set to promote the evaluation of technologies for annotation, indexing, classification and retrieval of multi-modal data, with the objective of providing information access to large collections of images in various usage scenarios and domains.
Task 1 - ImageCLEFmedical: participants will be requested to develop solutions for automatically identifying individual components from which captions are composed in Radiology Objects in COntext images and to combine natural language processing and computer vision for answering a medical question based on a visual image content.
Task 2 - ImageCLEFcoral: requires participants to automatically segment and label a collection of coral images that can be used in combination to create 3-dimensional models of an underwater environment.
Task 3 - ImageCLEFdrawnUI: requires participants to detect rectangular bounding boxes corresponding to website User Interface elements from hand-drawn and real-world images.
Task 4 - ImageCLEFaware: participants are required to provide automatic rankings of photographic user profiles in a series of real-life situations such as searching for a job, accommodation, insurance, bank loans, etc.
Lab Coordination: Bogdan Ionescu (University Politehnica of Bucharest), Henning Müller (University of Applied Sciences Western Switzerland), and Renaud Péteri (University of La Rochelle)
Twitter: @imageclef
8. LifeCLEF: Multimedia Life Species Identification
LifeCLEF lab aims at boosting research on the identification and prediction of living organisms in order to solve the taxonomic gap and improve our knowledge of biodiversity.
Task 1 - PlantCLEF: image-based plant species identification from herbarium sheets.
Task 2 - BirdCLEF: bird species identification from bird calls and songs in audio soundscapes.
Task 3 - GeoLifeCLEF: location-based species prediction based on environmental and occurrence data.
Task 4 - SnakeCLEF: image-based and location-based snake identification.
Lab Coordination: Alexis Joly (Inria, LIRMM), Henning Müller (HES-SO), and Hervé Goëau (CIRAD UMR AMAP)
9. LiLAS: Living Labs for Academic Search
In the LiLAS lab, we would like to bring together IR researchers interested in the online evaluation of academic search systems. The goal is to foster knowledge on improving the search for academic resources like literature, research data, and the interlinking between these resources.
Task 1 - Ad-hoc Search Ranking: the participants get access to the LIVIVO corpus that consists of about 80 million documents from more than 50 data sources in multiple languages from the Life Sciences.
Task 2 - Research Data Recommendations: the goal is to provide recommendations of research data that are relevant and related to the publication the user is currently viewing. The data for this task is taken from the academic search system GESIS Search from the Social Sciences.
Lab Coordination: Philipp Schaer (TH Köln - University of Applied Sciences), Johann Schaible (GESIS - Leibniz Institute for the Social Sciences), and Leyla Jael Garcia Castro (ZB MED - Information Centre for Life Sciences)
10. PAN: Digital Text Forensics and Stylometry
PAN is a series of scientific events and shared tasks on digital text forensics and stylometry, studying how to quantify writing style and improve authorship technology.
Task 1 - Cross-domain Authorship Verification: given two texts, determine if they are written by the same author.
Task 2 - Multi-Author Writing Style Analysis: given a text written by two or more authors, find all positions of writing style change.
Task 3 - Profiling Haters on Twitter: given a user's Twitter timeline, determine if they spread hate speech.
Lab Coordination: Martin Potthast (Leipzig University), Paolo Rosso (Universitat Politècnica de València), Efstathios Stamatatos (University of the Aegean), and Benno Stein (Bauhaus-Universität Weimar)
Twitter: @webis_de
11. SimpleText-202: (Re)Telling right scientific stories to non-specialists via text simplification
SimpleText aims at contributing to making science more open & accessible via automatic generation of simplified summaries of scientific documents.
Pilot Task 1 - Searching for background information: given a scientific text, provide background information from an external source to help a user to understand it (definitions, context,…).
Pilot Task 2 - Text Simplification: given a short scientific text (e.g. abstract), generate its simplified version.
Lab Coordination: Liana Ermakova, Patrice Bellot, Pavel Braslavski, Jaap Kamps, Josiane Mothe, Diana Nurbakova, and Eric San-Juan
Twitter: @SimpletextW
12. Touché: Argument Retrieval
Touché has been designed with the goal to establish a collaborative platform for researchers in the field of argument retrieval and to provide tools for developing and evaluating argument retrieval approaches.
Task 1 - Conversational Argument Retrieval: given a query on a controversial topic, the task is to retrieve relevant arguments from a focused crawl of online debate portals.
Task 2 - Comparative Argument Retrieval: given a comparative question, the task is to retrieve and rank documents from the ClueWeb12 that help to answer the comparative question with arguments and help to make an informed decision.
Lab Coordination: Alexander Bondarenko (Martin-Luther-Universität Halle-Wittenberg), Lukas Gienapp (Leipzig University), Maik Fröbe (Martin-Luther-Universität Halle-Wittenberg), Meriem Beloucif (Universität Hamburg), Yamen Ajjour (Martin-Luther-Universität Halle-Wittenberg), Alexander Panchenko (Skolkovo Institute of Science and Technology), Chris Biemann (Universität Hamburg), Benno Stein (Bauhaus-Universität Weimar), Henning Wachsmuth (Paderborn University), Martin Potthast (Leipzig University), and Matthias Hagen (Martin-Luther-Universität Halle-Wittenberg)
Twitter: @webis_de
*** ORGANIZERS ***
* General Chairs *
Bogdan Ionescu, University “Politehnica” of Bucharest, Romania
K. Selcuk Candan, Arizona State University, USA
* Program Chairs *
Henning Müller, University of Applied Sciences Western Switzerland, Switzerland
Lorraine Goeuriot, Université Grenoble Alpes, France
Birger Larsen, Aalborg University Copenhagen, Denmark
* Evaluation Lab Chairs *
Alexis Joly, INRIA Sophia-Antipolis, France
Maria Maistro, University of Copenhagen, Denmark
Florina Piroi, Vienna University of Technology, Austria
* Publicity Chairs *
Liviu Ștefan Daniel, University “Politehnica” of Bucharest, Romania
Mihai Dogariu, University “Politehnica” of Bucharest, Romania
* Outreach Program Chairs *
Yu-Gang Jiang, Fudan University, China - Asian Liaison
Hugo Jair Escalante, Instituto Nacional de Astrofisica, Optica y Electronica, Mexico - Central American Liaison
Fabio A. Gonzalez, National University of Colombia, Colombia - South American Liaison
Ben Herbst, Praelexis, South Africa - African Liaison
Abdulmotaleb El Saddik, University of Ottawa, Canada - North American Liaison
* Industry & Sponsorship Chairs *
Șeila Abdulamit, Vodafone, Romania
Mihai-Gabriel Constantin, University “Politehnica” of Bucharest, Romania
Bogdan Boteanu, University “Politehnica” of Bucharest, Romania
* Website & Social Media Chairs *
Denisa Ionașcu, University “Politehnica” of Bucharest, Romania
* Finance Chair *
Ion Marghescu, University “Politehnica” of Bucharest, Romania
* Proceedings Chairs *
Guglielmo Faggioli, University of Padua, Italy
Nicola Ferro, University of Padua, Ital