Evaluation demonstrators: OpCit Project

This page last updated

Evaluation

The Open Citation Project performed open, publicly announced evaluations of two principal demonstrators of reference linking and citation indexing services. A new service, Citebase, featured in the second evaluation, and on the basis of the results is a now a featured service of the arXiv eprint archives. The report on the evaluation of Citebase is the first detailed user evaluation of an open access Web citation indexing service.

Version Demonstrator First available Evaluation results

V. 2.1 Citation Analysis and Reference Linking demonstrator: Citebase March 2002 December 2002

V. 1.0 Reference Linking demonstrator May 2000 August 2000

Version	Demonstrator	First available	Evaluation results
V. 2.1	Citation Analysis and Reference Linking demonstrator: Citebase	March 2002	December 2002
V. 1.0	Reference Linking demonstrator	May 2000	August 2000

V. 2.1 Citation Analysis and Reference Linking demonstrator: Citebase (March 2002)

Find Cited and Linked full-text arXiv Documents (using Citebase, a citation-ranked search engine).
e.g. most-cited papers with Stephen Hawking as an author

(Almost) the whole archive - linked! With reference links and citation analysis.

Due to changes in the management of the underlying databases, V. 2.1 is a more robust and reliable version of the V. 2.0 Citation and Reference Linking demo, originally announced in August 2001.

Evaluation forms
- Form 1: Using Citebase, including sections about the evaluators, a practical exercise using Citebase, and user views of Citebase
- Form 2: User Satisfaction Questionnaire

Major results
- As exemplified by Citebase, Web-based citation indexing of open access eprint archives is closer to a state of readiness for serious use than had previously been realised
- Within the scope of its primary components, the search interface and services available from its rich bibliographic records, Citebase can be used simply and reliably for the purpose intended
- Citebase ompares favourably with other bibliographic services
- Tasks can be accomplished efficiently with Citebase regardless of the background of the user
- Better explanations and guidance are required for first-time users
- Coverage is seen as a limiting factor, even though Citebase indexes over 200,000 papers from arXiv: non-physicists were frustrated at the lack of papers from other sciences.
- Full evaluation report on Citebase

V. 1.0 Reference Linking demonstrator (May 2000)

Superseded and no longer fully operational, but still illustrates the principles of the original approach to reference linking used by the project.

Find Linked arXiv Documents (using a simple search interface)
Well-Linked Papers for non-physicists - examples of papers with lots of linked references

how and why

Results of V. 1.0 evaluation

Background to OpCit demonstrators

The Open Citation project is a research and development project began in October 1999 and completed in December 2003. In terms of features the V. 2.1 demonstrator exceeds the plan outlined in the original project proposal. It presents reference links and citation analysis of over 200,000 papers from the arXiv physics archives.

Evaluation of demonstrators produced by the project was based on a three-year evaluation plan. Not all the demonstrators envisaged within the original plan were produced, due to problems accessing content from other data providers at the time. Instead the project concentrated on extending the citation indexing services provided by Citebase.

An earlier demonstration of reference linking, evaluated by the Open Journal Project and reported in this paper, also involved a number of partners from OpCit.

We are grateful to our colleagues at arXiv.org, and the maintainers of the arXiv mirror at Southampton University, for permission to use their services in this project.