Evaluation of Citebase: a citation-ranked search service using the Open Archives Initiative

This exercise is designed to introduce you to a Web search service, Citebase, which allows you to discover science research papers by selecting different ranking criteria, such as citation impact and current usage. The aim of the exercise is to test how well the service serves users for the purpose of citation-ranked search.

Citebase covers selected full-paper archives that comply with the Open Archives Initiative (OAI). The largest of these archives is arXiv, now based at Cornell University.

The results of the evaluation will be published in the final reports of the Open Citation Project, which will be delivered to the JISC in the UK and the NSF in the USA, the joint funding bodies. These reports will be available on the Web during September. Most importantly, the results will inform further development of Citebase with a view to releasing it as a full OAI service in collaboration with arXiv. Queries about this exercise should be directed to Steve Hitchcock, sh94r@ecs.soton.ac.uk.

First, to put the results in some context, we need to know a little about you.

1 About you

1.1 Your subject interests. Are you a physicist, mathematician, computer scientsist, information scientist, other (specify)

1.2 Have you used the arXiv eprint archive before?
Yes, I use it daily
Yes, regularly
Yes, occasionally (less than monthly)
No

1.3 If you have used arXiv, which way do you access arXiv papers? (you may select one or more)
Daily or regular browsing
Email alerts from arXiv
Email alerts from other services
Reference links in papers
Bibliography or library services
Web search or other Web services
Don't use arXiv

Citebase is an Open Archives Initiative (OAI) service. This means it 'harvests' data from cooperating sites using the OAI Protocol for Metadata Harvesting.

1.4 Have you heard of the Open Archives Initiative? Y/N

1.5 Have you used any other OAI services or repositories? (you may select one or more)
Arc
EPrints.org
Kepler
My.OAI
Other OAI service (specify)

Comment on About you section

2 Practical exercise: building a short bibliography

You do not need to be a physicist to perform this exercise. Even if you are not, if Citebase is usable as intended you should be able to impress your colleagues with your detailed and up-to-date knowledge! We will focus on a selected topic in physics because that is the subject best served by arXiv. If you are a physcicist you can select another topic if you wish.

We are going to compile a short bibliography on the topic of string theory (or insert another physics topic of your choice). The bibliography will reveal the most-cited papers, the most prominent authors, the most popular papers and the latest papers on the topic. For evaluation purposes it would help us to know how long this exercise takes, so you will be asked to time yourself during this section.

Open the main Citebase search page in another browser window to begin the exercise. Using the services available from this page compile your bibliography based on answers to the following questions:

Start timing now

Arouna: Set separate boxes for each field in full reference: author(s), year, title and arXiv ID

2.1 Who is the most-cited author on string theory (or other topic)? Give full reference for one of his or her papers by copying data into the fields below
Arouna: check for 'witten' (not case sensitive)

2.2 Which paper on string theory is currently being used most often in arXiv? Copy reference data
Arouna: Check for 'what is string theory' (not case sensitive)

2.3 Which is the latest paper on string theory to be submitted to the arXiv? Give full reference
Arouna: Difficult to check correctness, possibly by date (i.e. posted in June 2002 and so includes '0206' in ID)

2.4 Which is the most-cited paper on string theory?  Copy reference data into fields
Arouna: Check for 'gauge theory correlators' (not case sensitive)

2.5 Which is the most highly cited paper that cites the most-cited paper above? Copy reference data
Arouna: Check for 'large N field theories string theory and gravity' (not case sensitive)

2.6 Co-cited articles - articles that are referred to in the same citing articles - are likely to be closely related. Which paper is most often co-cited with the most-cited paper above?
Arouna: Check for 'anti de sitter space and holography' (not case sensitive)

Arouna: the system is still not working properly and I haven't been able to complete this part yet
Take users to a paper co-cited with the most-cited paper. Ask a question that requires the full text. Then ask another question that follows an 'orange' link

Time taken to complete the exercise?      Stop timing now

Comment on the exercise

3 Your view of Citebase

3.1 In your view, which is the most useful feature of Citebase? search, ranking criteria, linked pdfs, other (specify)

3.2 What would most improve Citebase? More coverage, more papers, more topics, more links, better interface, better explanations, other (specify) (you may select one or more)

3.3 What services would you use to compile a bibliography in your own work and field? (you may select one or more)
My own bib list
Online library services
Offline library services
Web services
Other (specify)

3.4 Specify the name of the service you use most

3.5 If Citebase covered your subject to the degree it now covers physics, how might it compare with these bibliography services?
Very favourably, would probably switch to using Citebase
Favourably, might use Citebase in conjunction with existing services
Unfavourably, would not use Citebase

Comment on Citebase

Follow-up

We are seeking to announce an enhanced version of Citebase based on the findings of this evaluation. That enhanced version may also include links to journal services. If you would like to test that new version, or if you wish to see the results of this study, please select the appropriate boxes below and enter your email address:

I am interested in seeing the results of this study
I would be interested in participating in a follow-up exercise to test the addition of links to journal and library services
Your email address

Comment on this evaluation

There is one further part of this evaluation to complete, which will assess your level of satisfaction with Citebase. Tests have shown that this user satisfaction questionnaire, which you will find on the response page when you submit this form, should take no more than 3-4 minutes of your time.

SUBMIT THIS FORM button

Response to submit form

Congratulations! Your bibliography is printed below. With this you could pass for an expert on string theory!

More importantly, you have helped us assess the operation and usability of Citebase, which we hope to extend to other disciplines as more data becomes available.

The scenarios you have just performed have introduced you to ways of querying and browsing Citebase. You are invited now to complete a short user satisfaction form to give your initial impressions of using Citebase. This form is closely related to the first exercise and is an integral part of the overall evaluation.

Arouna: copy and adapt form from http://aims.ecs.soton.ac.uk/Pep.nsf/TargetUsabilityShort?OpenForm?

If you have any queries, you can email me from the link below.

Steve Hitchcock, Evaluation Coordinator
sh94r@ecs.soton.ac.uk

SUBMIT THIS FORM button

PRINT EVALUATOR'S BIBLIOGRAPHY HERE

Response to second submit form

Thank you. Your responses to the user satisfaction form have been received and will be included in results of the evaluation to be reported as part of the Open Citation Project. If you have signed up to receive notice of these results, or to participate in follow-up tests, we will contact you again. If not, we are most grateful for your participation.

If you have any queries, you can email me at the address below.

Steve Hitchcock, Evaluation Coordinator
sh94r@ecs.soton.ac.uk