View source for Conference and Labs of the Evaluation Forum

{{Infobox academic conference
| title = Conference and Labs of the Evaluation Forum
| logo = CLEF_Logo.png
| caption = Unlocking [[Information Access]]
| abbreviation = CLEF
| discipline = [[information retrieval]]
| publisher = [[Springer Science+Business Media|Springer]] and CEUR Workshop Proceedings
| history = {{start date and age|2000}} (formerly known as [[Cross-Language Evaluation Forum]] from 2000 to 2009)
| frequency = annual
| openaccess =
| website = {{URL|http://www.clef-initiative.eu/}}
}}
The '''Conference and Labs of the Evaluation Forum''' (formerly '''[[Cross-Language Evaluation Forum]]'''), or '''CLEF''', is an organization promoting research in multilingual [[information access]] (currently focusing on [[European Commissioner for Multilingualism|European languages]]). Its specific functions are to maintain an underlying framework for testing [[information retrieval]] systems and to create [[digital library|repositories]] of data for researchers to use in developing  comparable [[Technical standard|standards]].<ref name="Peters">{{cite conference | first1 = Carol | last1 = Peters| first2 = Martin | last2 = Braschler | first3 = Khalid | last3 = Choukri | first4 = Julio | last4 = Gonzalo | first5 = Michael | last5 = Kluck | title = The Future of Evaluation for Cross-Language Information Retrieval Systems | conference = Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001 | citeseerx = 10.1.1.109.7647 }}</ref>
The organization holds a conference every September in Europe since a first constituting workshop in 2000. From 1997 to 1999, [[Text Retrieval Conference|TREC]], the similar evaluation conference organised annually in the US, included a track for the evaluation of Cross-Language IR for European languages. This track was coordinated jointly by NIST and by a group of European volunteers that grew over the years. At the end of 1999, a decision by some of the participants was made to transfer the activity to Europe and set it up independently. The aim was to expand coverage to a larger number of languages and to focus on a wider range of issues, including monolingual system evaluation for languages other than English. Over the years, CLEF has been supported by a number of various EU funded projects and initiatives.
<ref name="clefhistory">Peters, Carol, and Martin Braschler. "European research letter: Cross‐language system evaluation: The CLEF campaigns." Journal of the Association for Information Science and Technology 52.12 (2001): 1067-1072.</ref>
CLEF 2019 marked the 20th anniversary of the conference and it was celebrated by publishing a book<ref name="clefbook">{{cite book | first1 = Nicola | last1 = Ferro| first2 = Carol | last2 = Peters | title = Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF | year = 2019 | publisher = Springer International Publishing, Germany | series = volume 41 of The Information Retrieval Series | volume = 41| doi = 10.1007/978-3-030-22948-1| isbn = 978-3-030-22947-4}}</ref> on the lessons learned in 20 years of evaluation activities.
==Structure of CLEF==
Before 2010,<ref>[http://clef.isti.cnr.it/ CLEF history before 2010]</ref> CLEF was organised as a workshop co-located with the [[European Conference on Digital Libraries]], consisting of a number of evaluation labs or tracks, similarly to TREC. In 2010, CLEF moved to become a self-sufficiently organised conference with evaluation labs, laboratory workshops, and a main conference track. In 2012, INEX, a workshop on retrieval and access to structured text, previously organised annually at [[Schloß Dagstuhl]], merged with CLEF to become one of its evaluation labs.
Prior to each CLEF conference, participants in evaluation labs receive a set of challenge tasks. The tasks are designed to test various aspects of information retrieval systems and encourage their development. Groups of researchers propose and organize campaigns to satisfy those tasks and the results are used as [[benchmark (computing)|benchmarks]] for the state of the art  in the specific areas.,<ref>{{cite journal | title = Special Issue on CLEF | journal = Information Retrieval | volume = 7 | issue = 1–2 | year = 2004 | doi = 10.1023/B:INRT.0000009472.59294.f6 }}</ref><ref>Fredric C. Gey, Noriko Kando, and Carol Peters "Cross-Language Information Retrieval: the way ahead" in ''Information Processing & Management''
vol. 41, no. 3,  p.415-431 May 2005, {{doi|10.1016/j.ipm.2004.06.006}}</ref>
In the beginning, CLEF focussed mainly on fairly typical information retrieval tasks, but has moved to more specific tasks. For example, the 2005 interactive image search task worked with illustrating non-fiction texts using images from [[Flickr]]<ref name="iclef">{{cite conference |
first1 = Jussi| last1 = Karlgren| author-link1 = Jussi Karlgren| first2 = Julio| last2 = Gonzalo| first3 = Paul|last3 = Clough|title= iclef 2006 overview: Searching the flickr www photo-sharing repository. | conference=Seventh Workshop of the Cross-Language Evaluation Forum CLEF 2006| year = 2006}}.</ref> and the 2010 medical retrieval task focused on retrieval of computed tomography,  MRI, and radiographic images.<ref name="ImageCLEFmed">{{cite web | last = Mueller| first = Henning| authorlink = | title = Medical Retrieval Task| publisher =ImageCLEF - Cross-language image retrieval evaluations | date = 20 May 2010| url =http://www.imageclef.org/2010/medical | format = | doi = | accessdate = 27 May 2010 }}</ref> In 2017, CLEF accommodated a number of tasks e.g. on identifying biological species from photographs or video clips, on stylistic analysis of authorship, and on health related information access.
==List of CLEF workshops and conferences==
{| class="wikitable"
|-
! Year
! Location
! Note
! Proceedings
! Editors
|-
| 2000
| [[Lisbon]]
| Constituting workshop
| Cross-Language Information Retrieval and Evaluation: Workshop of Cross-Language Evaluation Forum (CLEF 2000)
| Carol Ann Peters
|-
| 2001
| [[Darmstadt]]
|
| Evaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross—Language Evaluation Forum (CLEF 2001) Revised Papers
| Carol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
|-
| 2002
| [[Rome]]
|
| Advances in Cross-Language Information Retrieval: Third Workshop of the Cross—Language Evaluation Forum (CLEF 2002) Revised Papers
| Carol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
|-
| 2003
| [[Trondheim]]
|
| Comparative Evaluation of Multilingual Information Access Systems: Fourth Workshop of the Cross—Language Evaluation Forum (CLEF 2003) Revised Selected Papers
| Carol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
|-
| 2004
| [[Bath, Somerset|Bath]]
|
| Multilingual Information Access for Text, Speech and Images: Fifth Workshop of the Cross—Language Evaluation Forum (CLEF 2004) Revised Selected Papers
| Carol Ann Peters and Paul Clough and Julio Gonzalo and Gareth J F Jones and Martin Kluck and Bernardo Magnini
|-
| 2005
| [[Vienna]]
|
| Accessing Multilingual Information Repositories: Sixth Workshop of the Cross—Language Evaluation Forum (CLEF 2005). Revised Selected Papers
| Carol Ann Peters and Fred C Gey and Julio Gonzalo and Gareth J F Jones and Martin Kluck and Bernardo Magnini and Henning Müller and [[Maarten de Rijke]]
|-
| 2006
| [[Alicante]]
|
| Evaluation of Multilingual and Multi-modal Information Retrieval : Seventh Workshop of the Cross—Language Evaluation Forum (CLEF 2006). Revised Selected Papers
| Carol Ann Peters and Clough, P. and Fred C Gey and [[Jussi Karlgren]] and Bernardo Magnini and Doug Oard and [[Maarten de Rijke]] and M Stempfhuber
|-
| 2007
| [[Budapest]]
|
| Advances in Multilingual and Multimodal Information Retrieval: Eighth Workshop of the Cross—Language Evaluation Forum (CLEF 2007). Revised Selected Papers
| Carol Ann Peters and Jijkoun, V. and Mandl, T. and Henning Müller and Oard, D. W. and Peñas, A. and Petras, V. and Santos, D.
|-
| 2008
| [[Århus]]
|
| Evaluating Systems for Multilingual and Multimodal Information Access: Ninth Workshop of the Cross—Language Evaluation Forum (CLEF 2008). Revised Selected Papers
| Carol Ann Peters and Deselaers, T. and Nicola Ferro and Julio Gonzalo and Gareth J F Jones and Kurimo, M. and Mandl, T. and Peñas, A.
|-
| 2009
| [[Corfu]]
|
| Multilingual Information Access Evaluation Vol. I Text Retrieval Experiments—Tenth Workshop of the Cross—Language Evaluation Forum (CLEF 2009). Revised Selected Papers
| Carol Ann Peters and Di Nunzio, G. M. and Kurimo, M. and Mandl, T. and Mostefa, D. and Peñas, A. and Roda, G.
|-
| 2010
| [[Padua]]
| First conference
| Multilingual and Multimodal Information Access Evaluation. Proceedings of the International Conference of the Cross-Language Evaluation Forum (CLEF 2010
| [[Maristella Agosti]] and Nicola Ferro and Carol Ann Peters and [[Maarten de Rijke]] and [[Alan Smeaton]]
|-
| 2011
| [[Amsterdam]]
|
| Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011)
| Forner, P. and Julio Gonzalo and Kekäläinen, J. and Lalmas, M. and [[Maarten de Rijke]]
|-
| 2012
| [[Rome]]
|
| Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Proceedings of the Third International Conference of the CLEF Initiative (CLEF 2012)
| Catarci, T. and Forner, P. and Hiemstra, D. and Peñas, A. and Santucci, G.
|-
| 2013
| [[Valencia]]
|
| Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013)
| Forner, P. and Henning Müller and Paredes, R. and Rosso, P. and Stein, B.
|-
| 2014
| [[Sheffield]]
|
| Information Access Evaluation—Multilinguality, Multimodality, and Interaction. Proceedings of the Fifth International Conference of the CLEF Initiative (CLEF 2014)
| Kanoulas, E. and Lupu, M. and Clough, P. and Sanderson, M. and Hall, M. and Hanbury, A. and Toms, E.
|-
| 2015
| [[Toulouse]]
|
| Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Sixth International Conference of the CLEF Association (CLEF 2015)
| Mothe, J. and Savoy, J. and Jaap Kamps and Pinel-Sauvagnat, K. and Gareth J F Jones and SanJuan, E. and Cappellato, L. and Nicola Ferro
|-
| 2016
| [[Evora]]
|
| Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Seventh International Conference of the CLEF Association (CLEF 2016)
| Fuhr, N. and Quaresma, P. and Goncalves, T. and Larsen, B. and Balog, K. and Macdonald, C. and Cappellato, L. and Nicola Ferro
|-
| 2017
| [[Dublin]]
|
| Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eighth International Conference of the CLEF Association (CLEF 2017)
| Gareth J F Jones and Lawless, S. and Julio Gonzalo and Kelly, L. and Lorraine Goeuriot and Thomas Mandl and Cappellato, L. and Nicola Ferro
|-[http://clef.isti.cnr.it/ CLEF history before 2010]
| 2018
| [[Avignone]]
|
| Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018)
| Patrice Bellot, Chiraz Trabelsi, Josiane Mothe, Fionn Murtagh, Jian-Yun Nie, Laure Soulier, Eric SanJuan, Linda Cappellato, and Nicola Ferro
|
|-
| 2019
| [[Lugano]]
|
| Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019)
| Fabio Crestani, Martin Braschler, Jacques Savoy, Andreas Rauber, Henning Müller, David Losada, Gundula Heinatz, Linda Cappellato, and Nicola Ferro
|
|-
| 2020
| [[Thessaloniki]] (virtual due to COVID-19)
|
| Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eleventh International Conference of the CLEF Association (CLEF 2020)
| Avi Arampatzis, Evangelos Kanoulas, Theodora Tsikrika, Stefanos Vrochidis, Hideo Joho, Christina Lioma, Carsten Eickhoff, Aurélie Névéol, A., Linda Cappellato, and Nicola Ferro
|
|-
| 2021
| [[Bucharest]] (virtual due to COVID-19)
|
| Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021)
| K. Selçuk Candan, Bogdan Ionescu, Lorraine Goeuriot, Birger Larsen, Henning Müller, Alexis Joly, Maria Maistro, Florina Piroi, Guglielmo Faggioli, and Nicola Ferro
|}
==References==
{{reflist}}
== External links ==
* [https://www.clef-initiative.eu/ CLEF homepage]
* [http://clef.isti.cnr.it/ CLEF history before 2010]
{{Authority control}}
[[Category:Information retrieval organizations]]
{{Compu-conference-stub}}