This paper describes the research design and methodologies we used to assess the usefulness of MeSH (Medical Subject Headings) terms for different types of users in an interactive search environment. We observed four different kinds of information seekers using an experimental IR system: (1) search novices; (2) domain experts; (3) search experts and (4) medical librarians. We employed a user-oriented evaluation methodology to assess search effectiveness of automatic and manual indexing methods using TREC Genomics Track 2004 data set. Our approach demonstrated (1) the reusability of a large test collection originally created for TREC, (2) an experimental design that specifically considers types of searchers, system versions and search topic pairs by Graeco-Latin square design and (3) search topic variability can be alleviated by using different sets of equally difficult topics and well-controlled experimental design for contextual information retrieval evaluation.
|Title of host publication||International Workshop on Contextual Information Access, Seeking and Retrieval Evaluation (CIRSE) jointly with European Conference on Information Retrieval (ECIR)|
|Editors||L Tamine-Lechani L Tamine- Tamine-Lechani|
|Place of Publication||The Netherlands|
|Number of pages||6|
|Publication status||Published - 2010|
|Event||2nd CIRSE/ECIR International Workshop - Milton Keynes, UK, United Kingdom|
Duration: 28 Mar 2010 → …
|Conference||2nd CIRSE/ECIR International Workshop|
|Period||28/03/10 → …|
Liu, Y-H., & Wacholder, N. (2010). On search topic variability in interactive information retrieval. In L. T-L. L. T. Tamine-Lechani (Ed.), International Workshop on Contextual Information Access, Seeking and Retrieval Evaluation (CIRSE) jointly with European Conference on Information Retrieval (ECIR) (pp. 17-22). CEUR-WS.org.