Evaluation of Citebase: a citation-ranked search service using the Open Archives Initiative

The evaluation is now closed, thanks for your participation!

If you have further comments, please contact sh94r@ecs.soton.ac.uk

This exercise is designed to introduce you to a Web search service, Citebase, which allows you to find research papers and to display your results by selecting different ranking criteria, such as how many times the paper or the author has been cited or browsed. The aim of the exercise is to test how usable and useful you find this service.

Citebase covers selected full-paper archives that comply with the Open Archives Initiative (OAI). The largest of these archives is arXiv, now based at Cornell University.

The results of the evaluation will be published in the final reports of the Open Citation Project, which will be delivered to the JISC in the UK and the NSF in the USA, the joint funding bodies. These reports will be available on the Web during September. Most importantly, the results will guide further development of Citebase with a view to releasing it as a full OAI service in collaboration with arXiv. Queries about this exercise should be directed to Steve Hitchcock, sh94r@ecs.soton.ac.uk.


First, to put the results in some context, we need to know a little about you.

1 About you

1.1 Your subject interests. Are you a
Physicist    Mathematician    Computer scientist    Information scientist    Other (specify)

1.2 Have you used the arXiv eprint archive before?
Yes, I use it daily    Yes, regularly    Yes, occasionally (less than monthly)    No

1.3 If you have used arXiv, which way do you access arXiv papers? (you may select one or more)
Daily or regular browsing Email alerts from arXiv Email alerts from other services Reference links in papers
Web search or other Web services Bibliography or library services Don't use arXiv  Other (specify)

1.4 Had you heard of the Open Archives Initiative?      Yes      No  

1.5 Have you used any other OAI services? (you may select one or more)
Arc      Kepler     My.OAI      Other OAI service (specify)

Comment on About you section


2 Practical exercise: building a short bibliography

You do not need to be a physicist to perform this exercise. If Citebase is usable as intended you should be able to impress your colleagues with your detailed and up-to-date knowledge! We will focus on a specific topic in physics because that is well covered by arXiv.

We will now compile a short bibliography on string theory. The bibliography will reveal the most-cited papers, the most prominent authors and the most popular papers in arXiv. For evaluation purposes it would help us to know how long this exercise takes, so you will be asked to time yourself during this section.

Open the main Citebase search page in another browser window to begin the exercise. Using the services available from this page your bibliography will be compiled based on answers to the following questions:

Start timing now

2.1 Who is the most-cited (on average) author on string theory in arXiv? Copy the author's name into the field below
Author

2.2 Which paper on string theory is currently being browsed most often in arXiv? Copy the title of the paper into the field below
Title  

2.3 Which is the most-cited paper on string theory in arXiv?  Copy the title of the paper
 
Title

2.4 Which is the most highly cited paper that cites the most-cited paper above? Copy title
 
Title

2.5 Co-cited articles - articles that are referred to in the same citing articles - are likely to be closely related.
Which paper is most often co-cited with the most-cited paper above?
 
Title

Download a copy of the full-text of the most-cited paper on string theory.
2.6a What is the URL of the document you downloaded? 
URL

In this paper the author gives a reference to a paper co-authored with Strominger and Witten (ref [57]).
2.6b What is the title of the referenced paper?
Title

2.6c Did you use search to find the answer to 2.6b?  Yes  No
Stop timing now
Time taken to complete the exercise?   minutes                
Comment on the exercise in this section

3 Your view of Citebase

3.1 In your view, which are the most useful features of Citebase? (you may select one or more)                
Search Ranking criteria Linked pdfs Links to references
Links to citing papers Links to co-citing papers Citations graphs Other (specify)

3.2 What would most improve Citebase?                                        
Wider coverage More papers More links Faster results
Better interface Better explanations Other (specify)

3.3a  What services would you use to compile a bibliography in your own work and field? (you may select one or more)
My own bib list    Online library services    Offline library services    Web services    Other (specify)    

3.3b  Specify the name of the service you use most (if not own bib list)    

3.4 How does Citebase compare with these bibliography services (assuming, non physicists, that Citebase covered your subject to the degree it now covers physics)?


Comment on Citebase

4 Follow-up

We are seeking to announce an enhanced version of Citebase based on the findings of this evaluation. That enhanced version may also include links to journal services. If you would like to test that new version, or if you wish to see the results of this study, please select the appropriate boxes below and enter your email address:

I am interested in seeing the results of this study
I would be interested in participating in a follow-up exercise to test the addition of links to journal and library services

Your email address (Optional, you only need to include this if you have selected either of the boxes above)
Comment on this evaluation

AFTER THE SUBMISSION, THERE IS ONE FURTHER PART OF THIS EVALUATION TO COMPLETE, which will assess your level of satisfaction with Citebase. Tests have shown that this user satisfaction questionnaire, which you will find linked from the response page when you submit this form, should take no more than 3-4 minutes of your time.