| V1.0 demo evaluation
Open Citation Project - Reference Linking for Open Archives
V 1.0 demo: evaluation by the OpCit steering group: results
17 mails out; 13 sites
Period of evaluation: 19 July to 4 August 2000
Date of this report: 21 August 2000
Summary of author results and main findings.
(Mark selected answers with X where appropriate; you are welcome to elaborate)
How effective is this as a demonstration of
Very xxx 3
Moderately xxxx 4
Moderately. I don't like getting linked to PDF documents; I have to invest
a minute of download time to decide if I'm interested. I'd also like to
go to the journal, not to a preprint, when I have the option.
Moderately. The problem is, when I tried one of the papers (with SFX on)
and clicked on a red box, it said "A web browser has not been specified".
Going through the procedure didn't really help. Also, there were no links
in the text itself (orange boxes?) only red boxes at the end, which made
this a less effective demonstration.
I tend to say "very" when my comments re the SFX explanation are considered.
Also, I feel that ScreenCams could provide some additional value. But,
from my own experience with that, I feel that it would be great to have
something that would stream.
What features need to be improved?
I don't expect to be able to use a search facility when I'm linking directly
from a citation to an article. This is a different scale of things - e.g.
it's like trying to be an Ingenta or ISI.
Better search would be nice
I didn't see a search
Do you mean searching for links within a document? I'm confused -- I didn't
think there was any searching going on for users.
It is somewhat awkward to break an arXiv id into year, month, and article
number, it should be a single text entry box. If you stick with the current
scheme, the article number box should add leading zeros as needed. If you
go with a single box, the Los Alamos way of allowing current year or current
month to be inferred should be supported (i.e, entering a '1' should correspond
(today) to 0007001, entering a 6001 should be 0006001, etc.
So far, in the demo, it's really good. You know from looking at a link
if it will work or not. However, in future versions, I'd rather only see
the links that actually work, i.e., before making the link, make sure that
you can link to the target doc.
Journal links don't seem to work well
Didn't test this too extensively. However, picking a paper at random (OK,
it is one of mine), hep-th/9201076 all links have incorrect data (just
a year for preprints, no journal name for journal articles) - so links
don't work. Colors also seemed incorrect - for the two preprints cited,
they should be orange and not red since the arXiv ID wasn't there.
Link reliability: quite an issue, isn't it? Actually a very interesting
research matter on its own.
presentation is great. You might want to think about making the red boxes
into blue boxes, since blue is the default link color for most browsers,
but that's nitpicking.
a link button with a graphic would be more transparent to use
The orange and red boxes are acceptable, although my guess is that highlight
the link in red or orange rather than putting a box around it would look
I would really like to see the links available either within the PDF OR
separately as part of a wrapper page. Following link trails is inconvenient
if you have to download a full PDF every time rather than going from wrapper
page to wrapper page. Being able to extract links from a PDF into some
kind of a tagged XML format (including reference type and tagging the components
of the citation) would a very valuable tool. For the demo, having coded
boxes for reasons things don't link is fine. Presumably in a live version,
you would just not display anything for this references.
I am colour-blind -- like about 50% of the men on this planet -- and this
colour indication is really problematic for me. One could also think of
indicating these things with figures. Have you seen the way the CNRI people
show handle links on their own pages? Quite inspiring.
in our HTML articles we make internal article links. Links from a figure
mention ("See Fig. 1") to the figure, from a footnote mention to the footnote,
from a reference mention to the reference. You could try to make those
intra-document links, but let me tell you they're a lot harder than just
reference links. I think that reference links are probably the number one
bang-for-the-buck link you can put in scientific articles.
More links in the text would have been nice
link from main text to refs would be good
Links to SPIRES and ADS would be great.
addition of alternative resolvers i.e. other servers that provide service
links (see OpenURL) addresses the "more links" issue.
fine for me, but you should probably do some usability testing, if you
I'm not convinced that I found the SFX on/off feature very useful - it
just seemed to get in the way, but this probably says more about my lack
of appreciation of the "appropriate copy" problem.
I had problems with the user interface.
The SFX-type page should display detail of the object it is trying to find
and be a little more smart about what buttons are offered.
Should be a way to choose which arXiv mirror you go to. Links shouldn't
go from PDF to PDF. Much better to link to a wrapper page (link LANL abstract
page) that gives users more information (abstract, PDF size, etc.) before
final download decision. SFX page should give more information about the
two papers being linked - at least display the arXiv id's (but full reference
metadata would be even better). SFX base URL should be settable to point
to other resolvers. OpenURL spec should probably be adopted. SFX page could
use a lot of prettification. Adding a discovered arXiv id to reference
might be a nice feature.
I feel that your own SFX menu-screen could use some cosmetics. On http://arabica.ecs.soton.ac.uk/demo.html
, I suggest to put the header for the explanation of what SFX-style is
also on top of the page, not in the table. Plus, it would help to say --
in the explanation on SFX-style -- that "on" means the user will get a
list of extended services (including full-text links), while "off" means
that only links to full-text are provided. In the SFX explanation, the
sentence re "SFX in principle, not SFX as such" is very confusing. Why
not say that the provision of extended services via an intermediate menu-screen
fits into the SFX framework, but that in the demo you don't actually use
an ExLibris SFX server.
What features would you like to see added ? (rank from 1 highest)
Your highlighted demo PDF files contain bitmapped fonts rather than Type
1 fonts. This should be fixed. PDFs of my paper I downloaded were Type
Many of the PDFs are very poor quality (produced from scanned images rather
than Adobe Capture or directly from the origination program, I guess)
Inclusion of other linking services 8 points
Links to databases of secondary content 8 pts
Links to online journals 24 pts
Links to other archives 11 pts
Links to services/holdings in your institutional library 11 pts
How can the work of the project feed into your work?
All these should be handed off via virtual links such as LinkBaton.
I don't think you have to provide all of this yourself. Just open up your
environment via OpenURL, so that others can demonstrate their creativity
in creating linking services.
Links to online journals are our #1 concern. In terms of the demo, I don't
know what databases you might link to, but in our biomedical articles,
we have been asked to link genetic sequences to GenBank and some article
references to PubMed. These are both nice features, and will be required
at some point, but as I said above, the reference linking is most important
for now. The main thing that we would need added to the software before
we could use it would be much better linking to online journals. That includes:
A. recognizing the references to each journal, in spite of authors who
seem to be trying to make it difficult by using weird abbreviations for
B. algorithms for creating article links into each online journal. Our
journals all work the same, but many other journals don't. Since you can't
really take the time to test each link before you place it, you'll need
a list of online journals that you can link to and the method to turn volume,
issue, and page into a URL for each one.
C. possibly link status icons, based on journal info. For example: a free
journal would have a little icon next to links to it, notifying people
that they don't need a subscription to view the link. Subscriber-access
only journals might have a different icon (or no icon, since this is pretty
much the default :( ), and journals with pay-per-view might want to advertise
that in links.
We've done some work on A., B., and C., at least for our own journals
How would you like to see your work feed into the project?
We would like to run all of our PDFs through the software to generate reference
links to all of our other journals. Secondly, we'd love to link to other
publishers' journals and to PubMed. Additionally, we are looking into Manuscript
Submission and Review systems right now, where an author would submit a
PDF of his paper and the journal staff would review it online. If we could
link the references in that (totally randomly formatted) paper, it will
save reviewers tons of time in checking those references.
Its most direct relevance is a) to my work on DC-CITE, and b) to CrossRef.
If you can provide a simple PDF->PDF+OpCit converter that puts in SFX-type
links then I see no reason why we couldn't incorporate this in arXiv at
an early stage (we give two PDF options: PDF and PDF+OpCit links). If an
SFX-type server is used then there would presumably be no need to for us
to store any resource databases. We are currently testing OpenURL linking
from our abstract pages (http://sfx1.exlibris-usa.com/openurl/openurl.html).
It would be good to be able to specify a BASE-URL (taken from cookie) which
gives the user's chosen server as input to such a converter. However, I
imagine this creates a whole bunch of issues about whether you want to
use the type of ids/parameters specified by OpenURL. Our PDF usually has
links already: either within the document, to other arXiv documents, or
to URLs. I would like to see the internal links preserved as mentioned
The free-form reference parsing and dynamic adding of references to PDF
would be great for handling manuscripts within the editorial process (pre-production).
Even in production it would be helpful to add links to our PDFs (though
there is an argument against this - adding links is a dynamic process and
is viewed as an added-value service - making them part of the PDF rather
than a wrapper page means 1) a user looking at a saved PDF may not be getting
the most up-to-date links and 2) they are able to retain the added-value
service after their subscription lapses. SFX style linking addresses both
of these problems though (a user would have to go back to the publisher
gateway and this could be subscriber controlled). This is not necessarily
the dominant view around here - having links in the PDF is certainly convenient,
but I thought I would mention it.
we might be able to set up a service that would link PDFs for other people.
I would very much like to see CoRR hooked into this.
Support for either DOI linking or pragmatic citation linking (APS link
manager) for instance would be a valuable addition. APS could work to supply
accurate metadata for improved matching of Phys. Rev. citations
wouldn't it be nice to let users choose an SFX resolver. I can at least
think of 3: your system, our SFX server in Ghent and a demo SFX server
from Ex Libris. This approach would clearly illustrate the place of the
SFX framework in reference linking. And it would somehow make the mentioning
of Ghent and SFX as partners in OpCit more real. Implementing this possibility
only requires the implementation of the CookiePusher and the OpenURL (see
In most environments that we have dealt with so far, the work involved
is marginal. Initially -- to make things simple -- it could even start
by only providing OpenURLs for arXiv identifiers. As you can see from the
specs, the OpenURL is extremely simple in this case.
For most tex papers our current system uses hypertex to provide links within
papers and to other archive articles. Clearly your approach is more general
and will deal with the increasing number of non-tex submissions. I'd like
to see your system produce the intra-paper links from references to the
references section (though I imagine some people would prefer the reference
to go directly to the referred-to paper - actually less useful in my opinion
since in must cases one just wants to know which paper/worker is being
referred to). I see no reason why you should not create links to journals
and archives for references that cite both. Obviously, linking between
archives is relatively straightforward (though I notice that references
split by a line-break are not found, eg  in hep-th/0001001). I think
the real test is identifying references to journals and perhaps even books.
did you know arXiv now IS OpenURL-aware? The Ghent SFX-server already provides
quite some services for incoming OpenURL's carrying arXiv identifiers.
We fetch metadata back from arXiv using the OAi Dienst Subset.
I am quite interested to understand more about the technique you use to
actually insert links in the references: are the links being re-written
dynamically at every download, or do you have a link being inserted once
(in an batch process) that points at a front-end that connects to something
like a handle system for the real resolution of the link (the DOI-style
approach)? The more I work on these matters, the more I feel the latter
is a very interesting path. With this respect you may be interested in
our recent DOI-related experiment documented at http://sfxserv.rug.ac.be:8888/public/xref/
, and to be published as soon as I have time.
The OpCit Project
maintained by the Open Citation project. Contact