Improving Performance Support Systems through Information Retrieval Evaluation

New Search

Print Abstract

E-mail Abstract

Full Text

Save to My Collections

Export Citation

Schatz, S. (2006). Improving Performance Support Systems through Information Retrieval Evaluation. Journal of Interactive Learning Research, 17(4), 407-423. Chesapeake, VA: AACE.
Retrieved from http://www.editlib.org/p/20881.

• Share:
• 
• 
• 
• 
• 
• 

Journal Information

Journal of Interactive Learning Research
ISSN 1093-023X
Volume 17, Issue 4, October 2006
Association for the Advancement of Computing in Education (AACE)  Chesapeake, VA

More Information on JILR

Author

Steven Schatz, University of Hartford, USA

Abstract

This study examines existent and new methods for evaluating the success of information retrieval systems. The theory underlying current methods is not robust enough to allow testing retrieval using different meta-tagging schemas. Traditional measures rely on judgments of whether a document is relevant to a particular question. A good system returns all the relevant documents and no extraneous documents. There is a rich literature questioning the efficacy of relevance judgments. Such questions as, Relevant to whom? When? and To what purpose? are not well-answered in traditional theory. In this study, two new measures (Spink's Information Need and Cooper's Utility) are used in evaluating two search tools (tag-based and text-based), comparing these new measures with traditional measures and each other. The open-source Swish text-based search engine and a self-constructed tag-based search tool were used. Thirty-four educators searched for information using both search engines and evaluated the information retrieved by each. Construct measures, derived by multiplying each of the three measures (traditional, information need, and utility) by a rating of satisfaction were compared using two way analysis of variance. This study specifically analyzes small information systems. The design concepts would be untenable for large systems. Results indicated that there was a significant correlation between the three measures, indicating that the new measures provide an equivalent method of evaluating systems and have some significant advantages, which include not requiring relevance judgments and the ability to use the measures in situ.

Keywords

• Virtual Environments
• Information Sciences
• Educational Technology
• Learning Objects
• Information Communication Technologies

Also Read

• Topic Maps: Adopting User-Centred Indexing Technologies in Course Management Systems
• Making Efficient Use of Open Educational Resources Using a Multi-Layer Metadata Approach
• The Use of Learning Object Patterns and Metadata Vocabularies to Design Reusable and Adaptable Learning Resources.
• A “Simple Query Interface” Adapter for the Discovery and Exchange of Learning Resources
• A Framework for Concept-Based Digital Course Libraries
• Seven Constraints from One Limit: The inability of federated search tools to fully meet the informational needs of students
• The Use of Web 2.0 Social Networking Technologies to Expedite Collaboration Both in and Out of the University Classroom
• The Use of Weblogs in Learning and Assessment in Chinese Higher Education: Possibilities and Potential Problems
• Application of Frameworks in the Analysis and (Re)design of Interactive Visual Learning Tools
• Topic-Centered Aggregation of Presentations for Learning Object Repurposing

Tags