Technical Report ECSTR-IAM03-005 (ISBN: 0854 32798 3)
School of Electronics and Computer Science, University of Southampton

Evaluating Citebase, an open access Web-based citation-ranked search and impact discovery service

Steve Hitchcock, Arouna Woukeu, Tim Brody, Les Carr, Wendy Hall and Stevan Harnad

Open Citation Project, IAM Group, Department of Electronics and Computer Science, University of Southampton, SO17 1BJ, United Kingdom
Contact for correspondence: Steve Hitchcock

Version history of this report
Version 1.0, official report to JISC, released to selected users and evaluators December 2002
This version 2.0, edited for publication as a Technical Report, July 2003 (official TR format pdf; html)
Version 3.0, focus on usability results, edited for journal publication, draft, July 2003


Citebase is a new citation-ranked search and impact discovery service that measures citations of scholarly research papers which are openly accessible on the Web, i.e. papers that are assessable continuously online. Other services, such as ResearchIndex, have emerged in recent years to offer citation indexing of Web research papers. In the first detailed user evaluation of an open access Web citation indexing service, Citebase has been evaluated by nearly 200 users from different backgrounds. The paper details the procedures used in the evaluation, and analyses the results of this study, which took place between June and October 2002. It was found that within the scope of its primary components, the search interface and services available from its rich bibliographic records, Citebase can be used simply and reliably for the purpose intended, and that it compares favourably with other bibliographic services. It is shown tasks can be accomplished efficiently with Citebase regardless of the background of the user. More data need to be collected and the process refined before it is as reliable for measuring citation impact of indexed papers. Better explanations and guidance are required for first-time users. Coverage is seen as a limiting factor, even though Citebase indexes over 200,000 papers from arXiv. Non-physicists were frustrated at the lack of papers from other sciences. The principle of citation searching of open access archives has thus been demonstrated and need not be restricted to current users. Since the evaluation, Citebase has become a featured service of the ArXiv physics eprint archives.

View full text: Official TR format pdf; html