To be added
Add CC licence?
Box Highlight list of papers on correlation between downloads
and citations
dev countries
Box Review papers
? John Houghton, Open
Access – What are the economic benefits? A comparison of the United
Kingdom, Netherlands and Denmark (pdf 26pp)
Knowledge Exchange, 23 June 2009
Based on three report by the author: for JISC in the UK, for SURFin the
Netherlands, and for DEFF in Denmark (not yet available)
? John Houghton, Jos de Jonge, Marcia van Oploo, Costs and Benefits of
Research Communication: The Dutch Situation, SURFfoundation,
May 29, 2009
On examination these two Houghton reports do not seem to make direct
reference to the OA citation impact advantage in the calculations of
economic benefits
? Houghton, J. and Sheehan, P. (2009)
Estimating
the
Potential Impacts of Open Access to Research Findings
Economic Analysis and Policy
Journal, Vol. 39, No. 1, March 2009
http://www.eap-journal.com/
also available from http://ideas.repec.org/s/eap/articl.html
A great resource! Eloy Rodrigues, University of Minho, via
Twitter, Jul 17th 2009
Aneta OSTROWSKA
Open Access Journals
Quality – How to Measure It?
INFORUM 2009: 15th Conference on Professional Information Resources,
Prague, May 27-29, 2009
Asif-ul Haque, Paul Ginsparg (2009)
Positional Effects on Citation
and Readership in arXiv
arXiv.org, arXiv:0907.4740v1 [cs.DL], 27 Jul 2009
Davis blog comment : Expanding
and confirming an earlier study
on the positional effects in the arXiv
Joint, N. (2009)
The
Antaeus column: does the “open access” advantage exist? A librarian's
perspective
Library Review, Vol.
58, No. 7, 2009, 477-481
Greyson, D., Morgan, S., Hanley, G. and Wahyuni, D.
(2009)
Open access archiving and
article citations within health services and policy research
E-LIS, 14 Jul 2009, in Journal of
the Canadian Health Libraries Association (JCHLA) / Journal de
l'Association des bibliothèques de la santé du Canada
(JABSC), 2009, vol. 30, no. 2, 51-58
see also this poster
(1pp) with the same title, E-LIS, 14 Jul 2009, in Canadian Health Libraries Association/
Association des bibliothèques de la santé du Canada
(CHLA/ ABSC) Conference 2009, Winnepeg, Manitoba (Canada), May
30 - June 3, 2009
Comment on Evans and Riemer
Eisen, M., Letter to
the editor in Science, it is NOT junk blog, July 17,
2009: "the 8% statistic that Evans and Reimer highlight is
misleading. The authors’ supporting online material (figure S1C)
clearly shows that the impact of free access on citations is heavily
dependent on the age of the article at the time free access was
provided. In particular, when articles were made freely available
within 2 years of publication, their citations increased by almost 20%.
This
far more dramatic effect is the one scientists and journals should
consider when deciding when to provide free access. If this decision is
to be made purely on the basis of citation impact, the upward trend of
the curve in figure S1C argues strongly in favor of minimal delays."
Four Letters on this paper are published in the Science issue of 17 July 2009, Vol
325, issue 5938, including this one by Eisen (and Steven Salzberg), as well as others by Philip
M. Davis ('Increased Citations Not Guaranteed'), Alfred N. Burdett
('The Self-Selection Effect'), and Bruce Alberts ('The Sooner the
Better'), with a repsonse by the original authors.
Bernius, S. and Hanauske, M. extracts
On the citation network effects: "Applied to a network, the phenomenon
that “the
rich get richer and the poor get poorer” means that a
new vertex will be linked with a higher probability to
those existing vertices, which already have many
ingoing edges. In other words, an article which
has, at a particular time t, more citations than a
comparable article in the network, also has a higher
probability to get cited again in the future."
On the simulation method: "The simulation
program, written in Java and executable as a Java
applet, integrates the actors on the scientific publishing
market (scientists, libraries, publishers/journals) and
the interactions within and between these groups.
Market coordinating mechanisms are the reputation of
scientists and journals as well as the journals’ price and
usage (how often do scientists read articles of a
specific journal?)."
The following findings base upon simulation runs
with 200 authors, 9 journals (which publish up to 10
papers per period/issue), and 3 libraries. The
simulation duration is 100 periods. Assuming a journal
publishes six issues per year, within this setting
approximately 16 years would be simulated. It is
further assumed that all authors cite exactly 10 papers
out of the network when writing a new article.
In period t=20, 25 authors switch to the open access
strategy, meaning they deposit their papers (no matter
if accepted by a journal or not) additionally in a
repository, to which all authors in the network have
unrestricted access.
Results: Within the
framework of the used simulation specification, the
group of open access authors increased their average
rank from 100 in period t=20 to 68 at the end of the
simulation (average values of 50 simulation runs).
Some open access authors even manage to escape from
the bottom of the long tail – one also can say that they
escape from “"scientific invisibility"”.
Factors in the simulation results: the increase in citations of open
access papers in our simulation does not base on an
“"Early View Advantage"”. On the one hand, we assume
an accepted article to be published immediately in the
period of acceptance, on the other hand, self-archiving
is implemented in such a way that the publication in a
journal and the archiving of the article (if written by an
author belonging to the open access group) also fall
into the same period. ... the “"Selection Bias Postulate”"
(suggesting that the best authors are more likely to
make their articles available under an open access
model, and that they are more likely to do so with their
best work) is not directly implemented in the model.
The quality of an author (or more precisely, the
evolution of the quality of her articles over time) is
modelled as a stochastic element, and the selection of
those authors, which switch their strategy to open
access, is also completely randomized. Therefore, high
quality work is not overrepresented in the open access
repository.
In our model individual authors’ access to journals
is a driving force behind the open access citation
advantage.
This advantage over their colleagues certainly disappears when all
authors
switch to OA, but as
this is not to be expected in most disciplines in the medium term,
‘first movers’ can benefit
in the transition period."
Kurtz, M. and Brody, T. (2006)
The impact loss to authors
and research
e-Prints Soton, 12 July 2006
In Jacobs, N. (ed.), Open Access:
Key strategic, technical and economic aspects (Oxford, UK:
Chandos Publishing)
Mark Patterson on Mon,
PLoS Journals – measuring
impact where it matters
PLoS blog, 2009-07-13
On why PLoS is will no longer highlight the journal impact factor.
Instead it will present a range of metrics focussed on the published
paper, including individual citation counts from various sources,
blog and bookmark counts, links and searches, as illustrated
by Peter Binfield in the PLoS one community blog (March 31, 2009):
"rather than updating the PLoS Journal sites with the new numbers,
we’ve decided to stop promoting journal impact factors on our sites all
together. It’s time to move on, and focus efforts on more
sophisticated, flexible and meaningful measures."
Corbyn, Z. (2009)
Hefce
backs off citations in favour of peer review in REF
Times Higher Education, 18 June
2009
Comment: Jun 24, 2009 Stephen J Bensman JISC-Repositories list wrote:
"that
is pretty much how Garfield recommended citations should be used and
how they are used in US evaluations. You don’t use citations by
themselves but to balance your subjective judgments."
Stevan Harnad wrote: Gene is of course right that citations
alone are not and never were enough for research evaluation; they not
only need to be "balanced" against subjective (peer expert)
evaluations, but they need to be formally validated against them,
discipline by discipline.
Corbyn, Z.
Despite
losing their punch, some players still won big prizes
Times Higher Education, 19 March 2009
"Research impact was no guarantee of winning RAE
funding, study finds. Universities that saw some of the biggest
improvements in research funding in light of the 2008 research
assessment exercise actually experienced a decline in their overall
research performance in the years since the previous exercise in 2001."
Harnad blog comment Mar 19, 2009: "If the RAE peer panel raters'
criteria for ranking the universities varied or were inconsistent
between RAE 2001 and RAE 2008 then that is a problem with peer ratings
rather than with metrics (which, being objective, remain consistent)."
Jose
H. Canos Cerda, Manuel Llavador Campos and Eduardo Mena Nieto
What's
Wrong with Citation Counts?
D-Lib Magazine
, Volume 15 Number 3/4, March/April
2009
From the abstract: "We argue that a new approach based on the
collection of citation data at the time the papers are created can
overcome current limitations, and we propose a new framework in which
the research community is the owner of a Global Citation Registry
characterized by high quality citation data handled automatically."
Elsevier SciVal Spotlight, SciVal Funding
services http://info.spotlight.scival.com/,
http://info.funding.scival.com/
about Elsevier introduces SciVal: Performance planning and funding
solutions
http://libraryconnect.elsevier.com/lcn/0702/lcn070207.html
Laura Bowering Mullen
Increasing Impact of Scholarly Journal Articles: Practical Strategies
Librarians Can Share
Electronic Journal of Academic and Special Librarianship, Spring 2008
http://southernlibrarianship.icaap.org/content/v09n01/mullen_l01.html
add review section?
David Flaxbart, On Impact of OA, the Jury is Still
Out, Issues in Science and Technology Librarianship, Summer 2008
http://www.istl.org/08-summer/viewpoint.html
Suber OAN comment on Flaxbart 29 Aug 2008
"We desperately need objective, quantifiable evidence that OA does what
it claims to do, rather than taking these things as a matter of
near-religious faith." This leaves the impression that previous
claims that OA boosts citation impact are taken on faith, not grounded
in evidence. Flaxbart seems unaware of the
dozens of evidence-based studies
(links this bibliography) concluding that OA does indeed boost citation
impact. He doesn't mention them in his piece and or cite them in
his reference list. But he does note, correctly, that "[s]tudying
the effect of OA in the scholarly communication environment is
devilishly tricky." We're seeing multiple evidence-based
investigations taking on that devilish complexity. As in any
other domain, the investigators quarrel a bit about their methods and
interpretations of the data. But the debate is definitely
evidence v. evidence, not evidence v. faith.
Information Processing & Management, Volume 41, Issue 6, December
2005,
Pages 1441-1461, Available online 25 July 2005
OA?
Paul Huntington, David Nicholas, Hamid R. Jamali, Carol Tenopir
Article
decay in the digital environment: An analysis of usage of OhioLINK by
date of publication, employing deep log methods
Towards Usage-based Impact
Metrics: - First Results from the MESUR Project
arXiv.org, arXiv:0804.3791v1 [cs.DL], 23 Apr 2008
usage data has some distinct advantages over citation data. Most
importantly, usage data can be recorded
as soon as a scholarly artifact is made available online. It
therefore instantly refects the dynamics of the scientic process, as
opposed to citation data that is subject to significant
publication delays and therefore documents past developments. Also,
usage data can be recorded for a wide variety of accessible scholarly
artifacts, not just for journal
articles, and it can capture the actions of a user group that
significantly extends beyond the authors of journal articles
responsible for citations.
However, scholarly usage data and usage-based impact metrics have not
yet made inroads as reliable and community-accepted tools. The reasons
for this lack of acceptance are
manifold and include the absence of standards to record and
exchange usage data, uncertainties related to the compilation of usage
data sets that credibly represent global scholarly activity, and a
limited understanding of the nature of
usage based metrics as well of their exact meaning and applicability.
This paper has described preliminary results derived from
an analysis of a subset of the MESUR reference data set
that consists of over 200 million article-level usage events.
The results strongly point towards the imminent feasibility
of usage-based metrics of impact. The created journal usage
network surpasses the journal citation network in terms of
the number of data points from which it has been generated,
the density of its connections, the richness of its structure,
and its ability to model the many interdependencies that
characterize modern, interdisciplinary science. The rankings
according to a set of proposed usage-based metrics correlate
significantly with several citation-based metrics of impact,
including Thomson Scientific's journal Impact Factor, but
most strongly with network-based citation metrics that express
prestige, including betweenness centrality and PageRank.
PÉTER JACSÓ
The pros and cons of computing the h-index using Scopus. Online
Information Review, 32(4) 2008, 524-535
http://www.jacso.info/PDFs/jacso-pros-and-cons-of-computing-the-h-index-scopus.pdf
PÉTER JACSÓ
The pros and cons of computing the h-index using Web of Science. 32(5)
2008 (in press)
Link to CrossRef forward linking
http://www.crossref.org/02publishers/forward_linking_howto.html
Sophie L. Rovner
The Import Of Impact
Chemical & Engineering News, Volume 86, Number 20, May 19, 2008,
39-42
http://pubs.acs.org/cen/science/86/8621sci1.html
Tracey Caldwell, Numbers game hots up, Information World Review, 04 Feb
2008
http://www.iwr.co.uk/information-world-review/features/2208954/numbers-game-hots-3774199
James H. Fowler, Dag W. Aksnes
Does self-citation pay?
Scientometrics, Vol. 72, No. 3 (2007) 427-437
From the abstract: "We show that the more one cites oneself the
more one is cited by other scholars. Controlling for numerous sources
of
variation in cumulative citations from others, our models suggest that
each
additional self-citation increases the number of citations from others
by
about one after one year, and by about three after five years.
Moreover,
there is no significant penalty for the most frequent selfciters -
citation."
Henk F. Moed, New developments in citation analysis and research
evaluation, Information Services and Use 26, 2 (2006) pp. 135-137
http://iospress.metapress.com/(noq133rbspghqvaclol5pp45)/app/home/contribution.asp?referrer=parent&backto=issue,13,24;journal,1,28;linkingpublicationresults,1:103157,1
http://iospress.metapress.com/index/ETRNPPQJ55FLEC42.pdf
Metrics
Péter
Jacsó, SCImago Country Rank
Database, Péter's
Digital Reference Shelf, February 2009
Bornmann, L. & Daniel, H.-D. (2009). The state
of hindex research. Is the h index the ideal way to
measure research performance. EMBO Reports, 10(1), 2-6
Bornmann, L. & Daniel, H.-D. (2008). What
do citation counts measure? A review of studies on citing behavior. Journal
of Documentation, 64(1), 45-80.
Kurtz, Michael J.; Eichhorn, Guenther;
Accomazzi, Alberto; Grant, Carolyn S.;
Demleitner, Markus; Murray, Stephen S.;
Martimbeau, Nathalie; Elwell, Barbara
The
Bibliometric Properties of Article Readership Information (abs.
only)
Journal of the American Society for Information Science and Technology,
Vol. 56, p. 111, 2005
Matthew E. Falagas and three co-authors, Comparison of SCImago journal
rank indicator with journal impact factor, FASEB Journal, April 11,
2008. Only this abstract is free online
http://dx.doi.org/10.1096/fj.08-107938
Kosmopoulos et Pumain paper
Link Armbruster paper to metrics
Bornmann L, Mutz R, Daniel HD
Are There Better Indices for Evaluation Purposes than the h Index? A
Comparison of Nine Different
Variants of the h Index Using Data from Biomedicine
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY,
59(5), 830-837, 2008
http://www.lutz-bornmann.de/icons/BornmannMutzDanielFinal.pdf
Bornmann L, Mutz R, Neuhaus C, Daniel HD
Citation counts for research evaluation: standards of good practice for
analyzing bibliometric data and presenting and interpreting results
Ethics in Science and Environmental Politics, Vol. 8, Publication date:
February 12, 2008
http://www.int-res.com/abstracts/esep/pp7/
Long ref. list might be useful for more papers on metrics
J. E. Hirsch
Does the h-index have predictive power?
arXiv.org > physics > arXiv:0708.0646 [physics.soc-ph], 6 Aug
2007 (v1), 4 Oct 2007 (this version, v2)
http://arxiv.org/abs/0708.0646
Anne-Wil Harzing, Reflections on the h-index, Harzing.com, Second
version, 25 June 2007
http://www.harzing.com/pop_hindex.htm
MICHAEL JENSEN
The New Metrics of Scholarly Authority
The Chronicle of Higher Education, The Chronicle Review, Volume 53,
Issue 41, Page B6, June 15, 2007
http://chronicle.com/free/v53/i41/41b00601.htm
Rowlands, I; Nicholas, D
The missing link: journal usage metrics
ASLIB PROCEEDINGS 59 (3). 2007. p.222-228
Academics strike back at spurious rankings
D Butler, Nature 447, 514-515 (31 May 2007)
doi:10.1038/447514b
http://www.nature.com/nature/journal/v447/n7144/full/447514b.html
Harnad AmSci comment 3 Jun
see also comment on Webometrics Isidro Aguillo (Harnad fwd 5 Jun 2007)
Bornmann, L. & Daniel, H.-D.
What do we know about the h index?
Journal of the American Society for Information Science and Technology,
Volume 58, Issue 9, 2007, 1381-1385
http://www.lutz-bornmann.de/icons/BornmannDanielHReview.pdf
William Barendse
The strike rate index a new index for journal quality based on journal
size and the h-index of citations
Biomedical Digital Libraries, Biomedical Digital Libraries 2007, 4:3,
19 April 2007
http://www.bio-diglib.com/content/4/1/3
proposes a new impact measurement called the "strike rate index"
Sune Lehmann, Andrew D. Jackson and Benny E. Lautrup
Measures for measures
Nature 444, 1003-1004 (21 December 2006), published online 21 December
2006
http://www.nature.com/nature/journal/v444/n7122/full/4441003a.html
check if this is OA
Steele, C., Butler, L. and Kingsley, D. (2006)
The publishing imperative: the pervasive influence of publication
metrics
ANU Institutional Repository, 30 October 2006, also in Learned
Publishing, 19(4): 277-290, October 2006
Harnad blogs
"Metrics and Assessment" (Friday, April 14. 2006)
http://openaccess.eprints.org/index.php?/archives/79-guid.html
"Why is Southampton's G-Factor (web impact metric) so high?" (Thursday,
October 26. 2006)
http://openaccess.eprints.org/index.php?/archives/151-guid.html
Contemporary h-index
Antonis Sidiropoulos, Dimitrios Katsaros, and Yannis Manolopoulos,
Generalized h-index for disclosing latent facts in citation networks,
arXiv:cs.DL/0607066 v1 13 Jul 2006
http://arxiv.org/abs/cs.DL/0607066
aims to improve on the h-index by giving more weight to recent
articles, thus rewarding academics who maintain a steady level of
activity.
Institutional Repositories and Enhanced and Alternative Metrics of
Publication Impact: Report of an International Workshop held at
Humboldt University Berlin, 20-21 February 2006
detailed report
http://www.edoc.hu-berlin.de/series/dini-schriften/2006-8/PDF/8.pdf
also in High Energy Physics Libraries Webzine, Issue 13, October 2006
http://library.cern.ch/HEPLW/13/papers/2/
Egghe's g-index
Leo Egghe, Theory and practice of the g-index, Scientometrics, Vol. 69,
No 1 (2006), pp. 131-152
aims to improve on the h-index by giving more weight to highly-cited
articles
Anthony F.J. van Raan
Comparison of the Hirsch-index with standard bibliometric indicators
and with peer judgment for 147 chemistry research groups
arXiv.org, physics/0511206, 24 Nov 2005 (v1), last revised 24 Apr 2006
(this version, v2))
http://arxiv.org/abs/physics/0511206
J.E. Hirsch
An index to quantify an individual's scientific research output
arXiv.org, physics/0508025, 3 Aug 2005, last revised 29 Sep 2005 (this
version, v5), Proc.Nat.Acad.Sci., Vol. 102, No. 46, November 15, 2005,
16569-16572
http://arxiv.org/abs/physics/0508025
Need subsections on general metrics/methodologies as well as OA metrics?
Gideon Mann, David Mimno and Andrew McCallum, Bibliometric Impact
Measures Leveraging Topic Analysis. Joint Conference on Digital
Libraries (JCDL) Chapel Hill, NC, USA, June 11-15, 2006
http://www.cs.umass.edu/~mccallum/papers/impact-jcdl06.pdf
Steele, C., Butler, L. and Kingsley, D. (2006) The publishing
imperative: the pervasive influence of publication metrics
Add Mayr Constructing experimental indicators for Open Access documents
Bollen et al. Toward alternative metrics of journal impact: A
comparison of download and citation data
Noruzi, A. Toward alternative metrics of journal impact: A comparison
of download and citation data
Bollen, J., Rodriguez, M. A. and Van de Sompel, H. Journal Status
Stephen Pincock, UK plans research funding overhaul, The Scientist,
Wednesday 21 June, 2006
http://www.the-scientist.com/news/display/23683/
Comment by
Stevan Harnad, Let 1000 RAE Metric Flowers Bloom: Avoid Matthew Effect
as Self-Fulfilling Prophecy
http://openaccess.eprints.org/index.php?/archives/99-Let-1000-RAE-Metric-Flowers-Bloom-Avoid-Matthew-Effect-as-Self-Fulfilling-Prophecy.html
Colin Steele, Research with purpose, The Australian - Higher Education,
June 07, 2006
http://www.theaustralian.news.com.au/story/0,20867,19385536-12332,00.html
P. Chen, H. Xie, S. Maslov, S. Redner, Finding Scientific Gems with
Google, arXiv.org, physics/0604130, April 18, 2006
http://xxx.lanl.gov/abs/physics/0604130
Re: Future UK RAEs to be Metrics-Based
Stevan Harnad, 13 Apr 2006
http://listserv.utk.edu/cgi-bin/wa?A2=ind0604&L=sigmetrics&D=1&O=D&F=&S=&P=6443
H., F. Moed, Citation analysis in research evaluation. New York:
Springer, 2005
Mike Thelwall, Liwen Vaughan, Lennart Björneborn (2005).
'Webometrics'. Annual Review of Information Science and Technology, 39,
pp. 81-135
Lennart Björneborn and Peter Ingwersen (2004). "Toward a Basic
Framework for Webometrics". Journal of the American Society for
Information Science and Technology 55 (14): 1216-1227.
Ingwersen, P. The Calculation of Web Impact Factors, Journal of
Documentation, Vol. 54, No. 2, March 1998
http://projects.ics.hawaii.edu/~jacso/PDFs/ingwersen-calculation-of-WebIF.pdf
Tomas C. Almind and Peter Ingwersen (1997). "Informetric analyses on
the World Wide Web: Methodological approaches to 'webometrics'".
Journal of Documentation 53 (4): 404-426.
http://www.emeraldinsight.com/Insight/viewContentItem.do?contentType=Article&contentId=864059
Tools e.g.
The Hirsch index (h-index) for living chemists has been released.
The Royal Society of Chemistry summarizes the results in a press
release.
http://www.rsc.org/chemistryworld/News/2007/April/23040701.asp
Webometrics Ranking of World Universities (http://www.webometrics.info/)
See Aguillo clarification on AmSci 20 Mar 2007
The Webometrics Ranking of World Universities has created a new
Ranking of Repositories: http://www.webometrics.info/top200_rep.asp, 8
Feb 2008
see correspondence on subject vs central repositories and discussion on
criteria for ranking, AmSci and Sigmetrics lists, 8-11 Feb
26 May 2008 New Beta version of the Ranking Web of World Repositories
We have taken into account some of the suggestions regarding the
Ranking
Web of World Repositories for the new Beta 2 version. We have changed
the domain that it is now autonomous from the Universities Ranking one:
http://repositories.webometrics.info/
University Metrics http://www.universitymetrics.com/tiki-index.php
see also G-Factor - a new ranking methodology
http://www.universitymetrics.com/tiki-index.php?page=G-Factor
Eigenfactor http://www.eigenfactor.org ranks journals much as Google
ranks websites.
Center for Journal Ranking was launched in January 2007
http://www.journal-ranking.com/ranking/web/index.html
Carl Bergstrom, Eigenfactor: Measuring the value and prestige of
scholarly journals
C&RL News, Vol. 68, No. 5, May 2007
http://www.ala.org/ala/acrl/acrlpubs/crlnews/backissues2007/may07/Eigenfactor.htm
SCImago Journal & Country Rank
http://www.scimagojr.com/
End metrics
David Goodman RECENT MANUAL MEASUREMENTS OF OA AND OAA, liblicense 11
Jan 2006
Stevan Harnad Re: RECENT MANUAL MEASUREMENTS OF OA AND OAA,12 Jan 2006
Stevan Harnad and Chawki Hajjem, Re: Manual Evaluation of Algorithm
Performance on Identifying OA, AmSci, 28 Mar 2006, Goodman reply 30
March, also in Open Access Archivangelism, authro blog,
http://openaccess.eprints.org/index.php?/archives/76-Manual-Evaluation-of-Robot-Performance-in-Identifying-Open-Access-Articles.htm
I. "Meta-analysis of OA and OAA manual determinations." David Goodman,
, Kristen Antelman, and Nisa Bakkalbasi,
<http://eprints.rclis.org/archive/00005327/> (deposited 05
January 2006)
Stevan Harnad's group and our's have reported several manual
measurements in order to evaluate the accuracy of Chawki Hajjem's robot
program, which has been extensively used by Harnad's group. Our group
has now prepared an overall metaanalysis of the manual results.
II. "Evaluation of Algorithm Performance on Identifying OA." Kristin
Antelman, Nisa Bakkalbasi, David Goodman, Chawki Hajjem and Stevan
Harnad. <http://eprints.rclis.org/archive/00005326/>
(deposited 05 January 2006)
This is a signal-detection analysis of the accuracy of a robot in
detecting open access (OA) articles (by checking by hand how many of
the articles the robot tagged OA were really OA, and vice versa). We
found that the robot significantly overcodes for OA. In our Biology
sample, 40% of identified OA was in fact OA. In our Sociology sample,
only 18% of identified OA was in fact OA. Missed OA was lower: 12% in
Biology and 14% in Sociology. The sources of the error are impossible
to determine from the present data, since the algorithm did not capture
URL's for documents identified as OA. In conclusion, the robot is not
yet performing at a desirable level, and future work may be needed to
determine the causes, and improve the algorithm.
III. Goodman, David and Antelman, Kristin and Bakkalbasi, Nisa (2005)
Identifying Open Access Articles: Valid and Invalid Methods. In
Proceedings XXV Annual Charleston Conference: Issues in Book and Serial
Acquisition, Charleston, South Carolina.
http://dlist.sir.arizona.edu/968/
Many versions of an article are now visible on the web, including not
only open access (OA), but also paid access, preliminary versions,
abstracts, and mere references or citations. The purpose of OA requires
not only that the article can be read without the barrier of payment,
but also that the reader can find the article to be read without the
barrier of extensive searching. We will demonstrate the dubious
validity of one prominent system for identifying OA and measuring the
amount of OA and the OA Advantage. We will then briefly discuss the
remaining alternatives.
From Dr. David Goodman
Minor additions and updates 17 July 2009
Harnad blog comment on Gentil-Beccot
Harnad, S., OA
in High Energy Physics Arxiv Yields Five-Fold Citation Advantage,
Open Access Archivangelism blog, July 12. 2009: "This is an important
study, and most of its conclusions are valid (with caveats) ... From
the fact that when there is a
Green OA version available,
users prefer to consult that Green OA version rather than the journal
version, it definitely does not follow that journals are no
longer
necessary."
Add scintilla link to
Bollen PLOS ONE http://scintilla.nature.com/conversations?uri=10.1371/journal.pone.0004803
Castillo http://scintilla.nature.com/conversations?uri=http://www.ajnr.org/cgi/content/full/30/2/215
Evans,
J. A. and Reimer http://scintilla.nature.com/conversations?uri=10.1126/science.1154562
Eysenbach
http://scintilla.nature.com/conversations?uri=http://dx.doi.org/10.1371/journal.pbio.0040157
Bollen J, Van de Sompel H, Hagberg A, Chute R, 2009 A
Principal Component Analysis of 39 Scientific Impact Measures
now published in PLoS ONE 4(6):
e6022
http://dx.doi.org/10.1371/journal.pone.0006022
Loet
Leydesdorff
How are new
citation-based journal indicators adding to the bibliometric toolbox?
(full text requires subscription; abstract only)
Journal of the American Society for
Information Science and Technology, Vol. 60, No. 7, 2009, pp.
1327-1336, published online: 2 Feb 2009
From the abstract: "The launching of Scopus and Google Scholar, and
methodological developments in social-network analysis have made many
more indicators for evaluating journals available than the traditional
impact factor, cited half-life, and immediacy index of the ISI. In this
study, these new indicators are compared with one another and with the
older ones."
This is the latest version of this eprint
Harnad, S. (2009) Open Access Scientometrics and the
UK Research Assessment Exercise.
ECS EPrints, 27 Feb 2009, in Scientometrics, 79 (1), 2009, 147-156
http://dx.doi.org/10.1007/s11192-009-0409-z
http://eprints.ecs.soton.ac.uk/17142/
Added 15 July 2009
Kousha, K. and Abdoli, M. (2009)
The
citation impact of Open Access Agricultural Research: a comparison
between OA and Non-OA publications (pdf 12pp)
World Library And Information Congress: 75th IFLA General Conference
and Council, 23-27 August 2009, Milan, Italy
Blogged summary, Open
Access enhances accessibility and citation impact, International
Association of Agricultural Information Specialists, 13 July 2009: "The
results showed that there is an obvious citation advantage for
self-archived agriculture articles as compared to non-OA articles." -
"results indicate that self-archived research articles published
in the non-OA agriculture journals could attract nearly two times
more citations than their non-OA counterparts."
Gentil-Beccot, A., Mele, S., Brooks, T. (2009)
Citing and Reading Behaviours
in High-Energy Physics. How a Community Stopped Worrying about Journals
and Learned to Love Repositories
arXiv.org, arXiv:0906.5418v1 [cs.DL], v1, 30 Jun 2009
From
the abstract: The analysis of citation data demonstrates that free and
immediate online dissemination of preprints creates an immense citation
advantage in HEP, whereas publication in Open Access journals presents
no discernible advantage. In addition, the analysis of clickstreams in
the leading digital library of the field shows that HEP scientists
seldom read journals, preferring preprints instead.
Hodgkinson liblicense comment, Article on arXiv, 3 July 2009: "There is
a blip upwards in citations of about 0.15 citations per article per
month immediately upon publication, which does indicate some remaining
effect of journal publication, but this increase is gone within 12
months."
Lansingh, V. C. and Carter, M. J. (2009)
Does
Open Access in Ophthalmology Affect How Articles are Subsequently Cited
in Research? (abstract only, subscription required)
Ophthalmology, in press,
available online 22 June 2009 DOI:
http://dx.doi.org/10.1016/j.ophtha.2008.12.052 Scintilla
From the abstract: Examination of 480 articles in ophthalmology in the
experimental protocol and 415 articles in the control protocol. ...
Four subject areas were chosen to search the ophthalmology literature
in the PubMed database ... Searching started in December of 2003 and
worked back in time to the beginning of the year. The number of
subsequent citations for equal numbers of both open access (OA) and
closed access (CA) (by subscription) articles was quantified using the
Scopus database and Google search engine. A control protocol was also
carried out to ascertain that the sampling method was not
systematically biased by matching 6 ophthalmology journals (3 OA, 3 CA)
using their impact factors, and employing the same search methodology
to sample OA and CA articles. The total number of citations was
significantly higher for open access articles compared to closed access
articles for Scopus. However, univariate general linear model (GLM)
analysis showed that access was not a significant factor that explained
the citation data. Author number, country/region of publication,
subject area, language, and funding were the variables that had the
most effect and were statistically significant. Control protocol
results showed no significant difference between open and closed access
articles in regard to number of citations found by Scopus ... Unlike
other fields of science, open access thus far has not affected how
ophthalmology articles are cited in the literature.
Comment
Davis, P., Open Access
Not the Focus in Ophthalmology, Scholarly Kitchen blog, Jul
14, 2009: "this article
builds upon a growing wealth of evidence suggesting that the so-called
“open access citation advantage” is merely a spurious
relationship. It should serve as a gentle reminder
that correlation does not equal causation."
Lin, S.-K. (2009)
Full
Open Access
Journals Have Increased Impact Factors (editorial)
Molecules,
2009,
14(6):2254-2255
Mukherjee, B. (2009)
The hyperlinking
pattern of open-access journals in library and
information science: A cited citing reference study
Library & Information Science
Research, 31 (2), April 2009, 113-125
This paper appears to be another take on this
study
Bernius, S. and Hanauske, M. (2009)
Open
Access to Scientific Literature - Increasing Citations as an Incentive
for Authors to Make Their Publications Freely Accessible
Institute for Information Systems,
Frankfurt University, publications 2009
in 42nd Hawaii International
Conference on System Sciences (HICSS '09), 5-8 Jan. 2009, pp. 1-9
http://dx.doi.org/10.1109/HICSS.2009.335
Summary of results from also in
Bernius, S., Hanauske, M.,
König, W. and Dugall, B. (2009)
Open
Access
Models and their Implications for the Players on the Scientific
Publishing Market (see section 1.2)
Economic Analysis and Policy
Journal, Vol. 39, No. 1, March 2009
see extracts below
Frandsen, T. F. (2009)
The
effects of open access on un-published documents: A case study of
economics working papers
HAL: hprints-00352359, version 2, 12 January 2009, Journal
of Informetrics (2009) in press
Added 29 April 2009
Citation of biblio
"incredible resource"
OA Librarian blog, December 7, 2005 http://oalibrarian.blogspot.com/2005/12/citation-impact-bibliography-resource.html
JISC response to publishers on Houghton paper (Economic
implications of alternative scholarly publishing models:
Exploring the costs and benefits)
http://www.jisc.ac.uk/media/documents/publications/responseoneiaspmreport.pdf
(pdf 19pp)
Add journal citation
Lariviere, V., Gingras, Y. and Archambault, E. (2008)
The decline in the
concentration of citations, 1900-2007
Journal of the American Society for Information Science and Technology,
Vol. 60, No. 4, April 2009, 858 - 862, published online:
29 Jan 2009
Tiwari, A., Citation
Trend Line For PLoS Journals, Fisheye Perspective blog, April 25,
2009
A short illustrated blog on predicting the impact of a new journal. The
author, a bioscientist, evaluates two PLoS (OA) journals using Scopus
Journal Analyzer. Using the service's Trend Line and % Not Cited
parameters the author predicts that one, a new journal that doesn't yet
have an official impact factor, will soon rival the other, which does:
"I am sure it's impact factor (or quality or what ever you love) is
going to be same or may be much more." Does not claim to be
statistically sound.
Gargouri, Y. and Harnad, S.
Logistic
regression of potential explanatory variables on citation counts
Preprint 11/04/2009
Logistic regression
analysis on the correlation between citation counts (as dependent
variable) and a set of potential correlator/predictor variables.
Result: Published journal papers that are self-archived in
institutional repositories - in this study the repositories mandate
deposit, obviating the self-selection bias postulated by some to be a
factor in self-archiving - can achieve a citation advantage whether
published in journals of high and low impact: "Overall, OA is
correlated with a
significant citation advantage for all journal IF intervals".
Watson, A. B.
Comparing citations and
downloads for
individual articles
Journal of Vision, April 3, 2009 Volume 9, Number 4, Editorial i, Pages
1-4
Measures the correlation between downloads and citations counts for
articles in Journal of Vision: "Download statistics provide a useful
indicator, two years in advance, of eventual citations."
Comment: It's always disappointing to see data locked away in a
graph or a pdf or some similar digital or paper oubliette, there to
languish un(re)used. It's also disappointing to see a journal getting
way out ahead of the curve on something as important and valuable as
download metrics, and then missing an opportunity to continue to
innovate by providing real Open Data. http://www.sennoma.net/main/archives/2009/04/picking_on_j_vis.php
Johan Bollen, Herbert Van de Sompel, Aric Hagberg, Luis
Bettencourt, Ryan Chute, Marko A. Rodriguez, Lyudmila Balakireva (2009)
Clickstream
Data Yields High-Resolution Maps of Science
PLoS ONE 4(3): e4803, March 11, 2009
See also Nature news article
on this paper, 9 March 2009: "A striking difference in the usage maps
is that journals in the humanities and social sciences figure much more
prominently than in citation-based maps. The difference partly arises
because Bollen's study covers a wider literature than the citation
databases, which are biased towards natural sciences journals. "By
including practitioners we capture a much wider sample of the scholarly
community," adds Bollen. Usage maps are also more up to date than
citation ones because the
inherent delay in publication means it takes at least two years before
a paper will start to gather citations in sufficient numbers to be
meaningful. Anthony van Raan argues that this more current view may in
fact represent today's "fashions", rather than trends that will endure."
Davis comment blog Mar 16, 2009 http://scholarlykitchen.sspnet.org/2009/03/16/usage-map-of-science/
While the mapping of journal relationships using citations is
several
decades old, using clickstream data makes this article novel.
Åström, F. (2009)
Citation
patterns in open access journals
OpenAccess.se and the National Library of Sweden, February 25, 2009.
http://www.kb.se/Docs/about/projects/openaccess/citation_patterns_final090225.pdf
"Fewer analyses have investigated whether OA and non-OA journals in the
same research fields are citing the same
literature; and to what extent this reflects whether it is the same
kind (and thus
comparable) research that is published in the two forms of scholarly
publications. ... The citation structures in the journals were analysed
through MDS maps building on co-citation analyses, as well as a more
thorough comparison
investigating overlaps of cited authors and journals between the
different journals.
... The results of the analyses suggests that it is hard to draw any
overall conclusions on the matter of whether research published in OA
journals
is likely to have a larger citation impact or not."
This conclusion is unsurprising since the study did not measure impact
but mapped citation patterns between journals. It is suggested that
these mappings could improve understanding when comparing the impact of
OA and NOA journals.
Gaulé, P.
Access
to the
scientific literature in India
CEMI Working Paper 2009-004, February 23, 2009
Abstract: This paper uses an evidence-based approach to assess the
difficulties faced by developing country scientists in accessing the
scientific literature. I compare backward citations patterns of Swiss
and Indian scientists in a database of 43'150 scientific papers
published by scientists from either country in 2007. Controlling for
fields and quality with citing journal fixed effects, I find that
Indian scientists (1) have shorter references lists (2) are more likely
to cite articles from open access journals and (3) are less likely to
cite articles from expensive journals. The magnitude of the effects is
small which can be explained by informal file sharing practices among
scientists.
Castillo, M. (2009)
Citations and
Open Access:
Questionable Benefits
American Journal of Neuroradiology,
February 2009
An editorial.
(Ed. For the record, but I
cannot indicate what is questionable about OA, in this author's view,
as I can't access any part of this, not even an abstract.)
Mukherjee, B.
Do
open-access journals in library and information science have any
scholarly impact? A bibliometric study of selected open-access journals
using Google Scholar (full text requires subscription; abstract
only)
Journal of the American Society for Information Science and Technology,
Vol. 60, No. 3, March 2009, 581-594, published
online: 16 Dec 2008
From the abstract: "Using
17 fully open-access journals published uninterruptedly during 2000 to
2004 in the field of library and information science, the present study
investigates the impact of these open-access journals in terms of
quantity of articles published, subject distribution of the articles,
synchronous and diachronous impact factor, immediacy index, and
journals' and authors' self-citation."
There do not appear to be any
comparative findings (OA vs non-OA) revealed in the paper.
Added 26 February 2009
Evans, J. A. and Reimer, J. (2009)
Open Access and
Global Participation in Science (full text requires subscription;
summary only)
Science, Vol. 323. No. 5917,
20 February 2009, 1025
From the paper: "The influence of OA is more
modest than many have proposed, at c.8% for recently published
research, but our work provides clear support for its ability to
widen the global circle of those who can participate in science
and benefit from it."
See also Science podcast
interview with
James Evans
Articles on this paper: Dolgin, E., Online access
= more citations, The Scientist,
19th February 2009: "When the authors looked just at poorer countries,
however, they found that the influence of open access was more than
twice as strong. For example, in Bulgaria and Chile, researchers cited
nearly 20% more open access articles, and in Turkey and Brazil, the
number of citations rose by more than 25%. Free online availability "is
not a huge driver of science in the first world, but it shapes parts of
science in the rest of world," Evans told The Scientist."
Xie, Y., Open,
electronic access to research crucial for global reach, ars technica, February 19,
2009
Comment:
Harnad, S., Open
Access
Benefits for the Developed and Developing World: The Harvards and the
Have-Nots, Open Access Archivangelism blog, Feb 19, 2009:
"Evans & Reimer's study (E & R) is particularly timely and
useful. It shows that a large portion of the Open Access citation
impact advantage comes from providing the developing world with
access to the research produced by the developed world. Using a much
bigger database, E & R refute (without citing!) a
recent flawed study (Frandsen 2009) that reported that there
was no such effect ... Last, there is the question of the effect of
access embargoes. It is important to note that E & R's results
are not based on immediate OA but on free access after an embargo of up
to a year or more. Theirs is hence not an estimate of the increase
in citation impact that results from immediate Open Access; it is just
the increase that results from ending Embargoed Access."
See also this blog on the 'silly
spin' of the NSF press release on the paper and on the 'welter
of misunderstandings' of a follow-up blog based on the press
release in the Chronicle of Higher
Education.
Davis, P., Open
Access and Global Participation in Science, the scholarly
kitchen blog, Feb 19, 2009: "Advocates for open access will see this
article as supporting their cause. But those who spend time
reading the methodology will notice that message is not as
clear as the article implies."
Bollen, J., Van de Sompel, H., Hagberg, A. and Chute, R.
A principal component
analysis of 39 scientific impact measures
arXiv.org,arXiv:0902.2183v1 [cs.CY], 12 Feb. 2009
Comment
Davis, P., Scientific
Impact Measures Compared, the scholarly kitchen blog, Feb
17, 2009: "While this manuscript represents phenomenal
empirical work, “scientific impact” on philosophical grounds will
always remain a complex construct; and because of its complexity, it
will resist a single measure. We may all agree for practical
purposes that it be redefined with a new counting tool. But that
new tool is simply a different view of an enormous and complex beast."
Houghton, J., Rasmussen, B., Sheehan, P., Oppenheim, C., Morris, A.,
Creaser, C., Greenwood, H., Summers, M. and Gourlay, A. (2009)
Economic
implications of alternative scholarly publishing models:
Exploring the costs and benefits
JISC, 27 January 2009
Comment
Jacobs, N., Economic
case
for open access, JISC Information
Environment Team blog, 27 Jan. 2009: "The findings suggest that
there are both considerable cost savings to be made by the HE sector by
moving to open access, and significant benefits to the UK economy to be
gained by doing so. Both the potential cost savings and the benefits
run into hundreds of millions of pounds. ... a complex piece of work,
with potentially large implications"
Publishers'
joint statement, 13 Feb. 2009: "claims that if adopted universally
an exclusively open access business model would generate large savings
in the system costs for scholarly communication in the UK in our view
remain unproven. We expect that there will be a more detailed critique
in due course"
Norris, M. (2009)
The citation advantage of
open access articles
PhD thesis, Loughborough University Institutional Repository, 2009-01-15
Two published papers are based on this work (JASIST, ElPub)
Frandsen, T. F.
Attracted to open
access journals: a
bibliometric author analysis in the field of biology
Hprints, Nordic arts and humanities e-print archive,
HAL: hprints-00328270, version 1, 10 October 2008
also in Journal of Documentation,
January 2009
http://www.emeraldinsight.com/Insight/viewContentItem.do?contentType=Article&contentId=1766883
Comment
Davis, P., Open
Access: No Benefit for Poor Scientists,
the scholarly kitchen blog, Jan 14, 2009: "Open Access has a moral
agenda: to increase the flow of scientific information to researchers
in developing nations. Yet a new study suggests that authors in
developing countries are no more likely to write papers for Open Access
journals and are no more likely to cite Open Access articles." Appended
to Davis' blog, Gaule, P., 14 Jan. 2009: "Ms Frandsen’s conclusion that
‘authors from developing countries do not cite open access more than
authors from developed countries’ is not based on solid evidence. While
she reports the p-value and not the standard errors, it is clear from
her regression results that she cannot statistically rule out the
possibility that authors from developing countries may be more likely
to cite open access journals."
Harnad, S., Comparing
OA/non-OA in Developing Countries, Open Access Archivangelism blog,
January 14. 2009: "The Frandsen
study focused on OA journals, not on OA articles. It is
problematic to compare OA and non-OA journals, because journals differ
in quality and content, and OA journals tend to be newer and fewer than
non-OA journals (and often not at the top of the quality hierarchy). In
contrast, most studies that have compared OA and
non-OA articles within the same journal and year have found a
significant citation advantage for OA. It is highly unlikely that this
is only a developed-world effect; indeed it is almost certain that a
goodly portion of OA's enhanced access, usage and impact comes from
developing-world users."
Chan, L., Comparing
OA/non-OA in Developing Countries, Open Access Archivangelism blog,
January 14. 2009: "The Frandsen study focuses on biology
journals and I am not sure what percentage of them are available to DC
researchers through HINARI/AGORA. This would explain why researchers in
this area would not need to rely on OA materials as much. Citation
behaviour is complex indeed and more studies on OA's impact in the
developing world are clearly needed. Davis's eagerness to
pronounce that there is "No Benefit for Poor Scientists" based on one
study is highly premature."
Frandsen, T. F.
The integration
of open access journals in the scholarly communication
system: Three science fields
Hprints, Nordic arts and humanities e-print archive,
HAL: hprints-00326285, version 1, 2 October 2008
also in Information Processing &
Management, January 2009
http://dx.doi.org/10.1016/j.ipm.2008.06.001
From the abstract: "This study is an analysis of the citing behaviour
in (open access) journals within three science fields:
biology, mathematics, and pharmacy and pharmacology. The integration of
OAJs in the scholarly communication
system varies considerably across fields. The implications for
bibliometric research are discussed."
Cross, J.
Impact
factors – the basics
The E-Resources Management Handbook (2006 - present), UKSG, this
chapter published online:
03 February 2009
Added 13 January 2009
Norris, M., Oppenheim, C., and Rowland, F.
The
citation advantage of open-access articles (full text requires
subscription; abstract only)
Journal of the American Society for
Information Science and Technology
ADDED also available from Loughborough University Institutional
Repository, 2009-01-12 http://hdl.handle.net/2134/4083
Dr. Michael Norris has been named Highly Commended Award
winner of the 2008
Emerald/EFMD Outstanding Doctoral Research Award in the
Information Science category for his doctoral thesis 'The citation
advantage of open access articles', on which this publication is
based.
Norris, M., Oppenheim, C. and Rowland, F.
Finding open
access articles using Google, Google Scholar, OAIster and OpenDOAR
Online Information Review, Vol. 32, No. 6, 2008, 709 - 715
also available from Loughborough University Institutional Repository,
2009-01-12 http://hdl.handle.net/2134/4084
From the abstract: "Google, Google Scholar, OAIster and OpenDOAR were
used to
try to locate OA versions of peer reviewed journal articles drawn from
three subjects
(ecology, economics, and sociology). The paper shows the relative
effectiveness of the search tools in
these three subjects. The results indicate that those wanting to find
OA articles in
these subjects, for the moment at least, should use the general search
engines Google
and Google Scholar first rather than OpenDOAR or OAIster."
Gingras, Y., Larivière, V. and Archambault, E.
Literature Citations in the Internet Era (Letter)
Science, Vol. 323, No. 5910, 2
January 2009, 36
"(Evans's)
conclusions
are not warranted by (the) data. ... In
fact, Evans's conclusions only reflect a transient phenomenon related
to recent access to online publications and to the fact that the method
used does not take into account time delays between citation year and
publication year."
O'Leary, D. E.
The relationship
between citations and number of downloads (subscription)
Decision Support Systems, Vol.
45, No. 4, November 2008, 972-980,
available online 11 April 2008
Broadly agrees with earlier findings (e.g. Brody) about the correlation
- 'strong positive statistically significant relationship' - between
downloads and citations for digital papers, notably for the
most-downloaded, 'top' papers, in this case based on data for a single,
focussed source, the journal Decision
Support Systems
De Groote, S. L.
Citation
patterns of online and print journals in the digital age
J. Med. Libr. Assoc., 2008 October; 96(4): 362–369.
http://dx.doi.org/10.3163/1536-5050.96.4.012
Pringle,
J.
Trends in the use
of ISI citation databases for
evaluation
Learned
Publishing, Vol. 21, No. 2, April 2008, 85-91
Abstract: This paper explores the factors shaping the current uses of
the ISI
citation databases in evaluation both of journals and of individual
scholars and their institutions. Given the intense focus on outcomes
evaluation, in a context of increasing 'democratization' of metrics in
today's digital world, it is easy to lose focus on the appropriate ways
to use these resources, and misuse can result.
Added 24 November 2008
Tenopir, C. and King, D. W.
Electronic Journals and Changes in
Scholarly Article Seeking and Reading Patterns
D-Lib Magazine, Vol. 14 No.
11/12, November/December 2008
Harnad, Davis blog comments
Harnad: "confirmation of the finding (of Kurtz and
others) -- that as more articles become accessible, more articles are
indeed accessed (and read), but fewer articles are cited (and those are
cited more) -- is best explained by the increased selectivity made
possible by that increased accessibility ... Open Access (OA) allows
all the cream to rise to the top; accessibility is no longer a
constraint on what to cite. (One of the reasons the top articles are
more likely to be made OA is precisely that they are also more likely
to be cited more if they are made OA!)
Davis: "What is clear from their data is that scientists are reading
more articles from more journals. On face value, this would seem
to
imply that scientists would be citing a greater diversity of
articles.
On the other hand, Tenopir and King also report that scientists are
using searching tools, citations, and people more often to help them
decide what to read. This would imply that scientists are
becoming
more focused in their information seeking behavior, which would lead to
a reduction of citation diversity."
Gaule, P. and Maystre, N.
Getting
cited: does open access help?
Ecole Polytechnique Fédérale de Lausanne,
CEMI-WORKINGPAPER-2008-007, November 2008
Davis blog comment Nov 18, 2008
Cho, S.-R.
New evaluation
indexes for articles and authors’ academic achievements based on Open
Access Resources (subscription)
Scientometrics, Vol. 77, No. 1
(2008) 91-112, published online: 24 July 2008
Levitt, J. M. and Thelwall, M.
Added 13 November 2008
Singleton, A., Learned Publishing, Vol. 21, No. 4, October 2008,
329-331
"For anyone experienced in
these matters, or even in scholarly
publishing or library and information
science (LIS) generally, it consists
mainly of statements of the fairly
obvious - and a lot of repetition.
...
Overall this is a curate's egg of a
report. I think we deserve better.
Since it comes in the name of three
prestigious institutions, one might
have expected the review to be more
thorough and, perhaps, the text to
be better edited, but for policymakers
the examples and simplicity
of statements will, I think, make it a
useful document.
http://dx.doi.org/10.1087/095315108X356761
more on Davis BMJ
Harnad, S. ,
On Eggs and Citations, August 29. 2008
Failing to observe a platypus laying eggs is not a demonstration
that the platypus does not lay eggs. ... Failing to observe a
significant OA citation Advantage within a year of publication (or a
year and a half -- or longer, as the case may be) with randomized
OA does not demonstrate that the many
studies that do observe a significant OA citation
Advantage with nonrandomized OA are simply
reporting self-selection artifacts (i.e., selective provision
of OA for the more highly citable articles.) ... The many reports of
the nonrandomized OA Citation Advantage are based on samples
that weresufficiently large, and on a sufficiently long time-scale
(almost never as short as a year) to detect a significant OA Citation
Advantage. A failure to observe a significant effect with
small, early samples, on short time-scales -- whether randomized or
nonrandomized -- is simple that: a failure to observe a significant
effect: Keep testing till the size and duration of your sample of
randomized and nonrandomized OA is big enough to test your
self-selection hypothesis (i.e., comparable with the other studies that
have detected the effect).
http://openaccess.eprints.org/index.php?/archives/452-On-Eggs-and-Citations.html
See also Fiona Godlee,
F., Open access to research Increases readership but not citations
(editorial), BMJ 2008;337:a1051, published 31 July 2008,
http://www.bmj.com/cgi/doi/10.1136/bmj.a1051
"This week the BMJ publishes a paper that has nothing directly
to do with medicine or health care. It does, however, have everything
to do with access to research results, a topic that should interest
authors and readers in any field. ... questions that reach to the very
heart of the way in which scientists and clinicians communicate. ...
questions also reach to the heart of how academics are acknowledged and
rewarded.
Change Davis URL http://www.bmj.com/cgi/doi/10.1136/bmj.a568
Added 10 November 2008
Kousha,
K. (2008)
Proceedings of WIS 2008, Fourth
International Conference on
Webometrics, Informetrics and Scientometrics & Ninth COLLNET
Meeting (Berlin, 28 July - 1 August 2008), edited by H.
Kretschmer and
F. Havemann, October 2008
Lariviere, V., Gingras, Y. and Archambault, E. (2008)
The decline in the
concentration of citations, 1900-2007
arXiv.org,
arXiv:0809.5250v1 [physics.soc-ph], 30 Sep 2008
From the abstract: "This paper challenges recent research (Evans, 2008)
reporting that the concentration of cited scientific literature
increases with the online availability of articles and journals. ...
contrary to what was reported by Evans, the dispersion of citations is
actually increasing."
Davis comment 13 October 2008 The Scholalry Kitchen blog, http://scholarlykitchen.sspnet.org/2008/10/13/citation-controversy/
"a
new analysis taking aim at (Evans') diversity claim. ...
Unlike Evan’s article,
this paper does not require knowledge of negative binomial regression,
or any advanced statistics for that matter; and because of the
simplicity and descriptive approach to their analysis, it is very
convincing. Granted, Evans is using a different approach, looking
at the effect of when journals became available online on citation
behavior and whether commercial access or free access changes the
outcome. For that reason, we should not discount the merit of
Evan’s paper. ...
What makes this controversy interesting is that both studies make
theoretical sense. Like
many scientific controversies, the argument over citation diversity
will move toward consensus and closure. For the meantime, the
debate remains open."
Henneken, E. A., Kurtz, M. J., Accomazzi, A., Grant, C. S., Thompson,
D., Bohlen, E. and Murray, S. S. (2008)
Use of Astronomical Literature
- A Report on Usage Patterns
arXiv.org, arXiv:0808.0103v1 [cs.DL], 1 Aug 2008
Dietrich, J. P. (2008)
Disentangling
visibility and self-promotion bias in the arXiv: astro-ph positional
citation effect
arXiv.org, arXiv:0805.0307v2 [astro-ph], 25 Jun 2008, in Publications
of the
Astronomical Society of the Pacific,120
(869): 801-804
Harnad blog comment 5 Aug 2008: The authors rightly point out that in a
high-output field like
astrophysics, visibility is an important factor in usage and citations,
and authors need alerting and navigation aids based on importance,
relevance and quality, rather than on random timing and author
self-promotion biasses.
Suber, PS: Also see my 2005 article, Visibility
beyond open access: "I would add that in fields -- whether high- or
low-output -- that,
unlike astrophysics, are not yet OA, accessibility itself probably has
much the same sort of effect on citations that visibility does in an OA
field like astrophysics. (Even maximized visibility cannot make
articles accessible to those who cannot afford access to the
full-text.)"
Davis blog comment 7 Aug
http://scholarlykitchen.sspnet.org/2008/08/07/the-importance-of-being-first/
Oppenheim, C. (2008)
Journal of Librarianship and
Information Science, 40 (3): 147-149, September
2008
Oppenheim, C. and Summers, M. A. C.
Citation counts
and the Research Assessment Exercise, part VI: Unit of assessment 67
(music)
Information Research, 13(2),
paper 342, June 2008
Added to Citation analysis, indexes
and impact
factors, 10 Nov.
Leydesdorff, L. (2008)
How
are new citation-based journal indicators adding to the bibliometric
toolbox?
Author preprint, undated, (announced 31 Oct 2008)
Radicchi, F., Fortunato, S. and Castellano, C. (2008)
Universality of citation
distributions: towards an objective measure of scientific impact
arXiv.org,
arXiv:0806.0974v2 [physics.soc-ph], 5 Jun 2008 (v1), last revised
27 Oct 2008
Davis
blog comment, Universal Citations, Nov 3, 2008: differences
between disciplines can be quickly remediated by a simple, intuitive
calculation: divide the number of citations to a paper by the
average number of citations to all papers in its discipline for that
year. The effect is stunning, and seems to hold irrespective of
the publication year studied. Looking at the effect of this
normalization is like looking at ducks lining up in a row." http://scholarlykitchen.sspnet.org/2008/11/03/universal-citations/
Banks, M. A. and Dellavalle,
R. (2008)
Emerging
Alternatives to the Impact Factor
E-LIS, 05 September 2008, also in OCLC
Systems & Services, 24(3)
Brumback, R. A. (2008)
Worshiping false idols: the impact factor dilemma
J. Child Neurol., Vol. 23, No. 4, April 2008, 365-367
http://jcn.sagepub.com/cgi/reprint/23/4/365
"the opacity in Thomson Scientific’s refusal to reveal the details of
their calculations only serves to increase suspicion about possible
data manipulations. ... Now would seem to be the appropriate time for
the academic community to demand valid metrics to assess published
scientific material"
Comment on Brumback: Pringle J. "Worshiping False Idols: The
Impact Factor Dilemma":
Correcting the Record. J Child Neurol Vol. 23 No. 9, September
2008 2008 23: 1092.
http://jcn.sagepub.com/cgi/reprint/23/9/1092
"reiterates statements made by Michael Rossner about a supposed
discrepancy
in our database relating to the impact factor calculations
for the Journal of Experimental Medicine. The Rossner
editorial, though republished several times, each time
repeats the same discredited assertions. I refer your readers
to the corrections published by Thomson Scientific
Brumback R. A. Response to Correspondence, "`Worshiping False
Idols:
The Impact Factor Dilemma': Correcting the Record." J Child Neurol
2008
"it is disappointing that Pringle chose to use typical faulty reasoning
by attacking my citing of the article by Rossner et al (which
was just 1 of the total 32 references) rather than addressing the real
issues raised in my editorial. ... Concerns about the journal impact
factor values are not new and have been voiced for more than a decade
(but mostly to deaf ears)8-38 Although it would be
preferable to have journal
indicators produced by an independent not-for-profit organization, the
Thomson Scientific journal impact factor could be
acceptable if Thomson Scientific were carefully to address each
specific concern that has been raised and then provide
a truly open transparent reporting system with resultant nonmanipulable
values."
Added 25 September 2008
Davis, P. M.
Author-choice open access
publishing in the biological and medical literature: a citation analysis
arXiv.org, arXiv:0808.2428v1 [cs.DL], 18 Aug 2008
forthcoming in the Journal of the American Society for Information
Science & Technology
This study is a follow-up to a controlled
trial of open access publishing published in BMJ
"According to a study of 11 biological and medical journals that allow
authors the choice of making their articles freely available from the
publisher’s website, few show any evidence of a citation advantage. For
those that do, the effect appears to be diminishing over time.
...analyzed over eleven thousand articles published in journals since
2003, sixteen hundred of these articles (15%) adopting the
author-choice open access model."
see blog entry (Aug 19, 2008) and responses http://scholarlykitchen.sspnet.org/2008/08/19/oa-advantage-deminishin/
Harnad response, also at
http://openaccess.eprints.org/index.php?/archives/451-guid.html,
includes line-by-line commentary/critique
"The outcome, confirming previous studies (on both paid and unpaid OA),
is a significant OA citation advantage, but a small one (21%, 4% of it
correlated with other article variables such as number of authors,
references, and pages). The author infers that the size of the OA
advantage in this biomedical sample has been shrinking annually from
2004-2007, but the data suggest the opposite."
Clauson, K. A., Veronin, M. A., Khanfar, N. M. and Lou, J. Q.
Open-access
publishing for pharmacy-focused journals
American Journal of Health-System Pharmacy, August 15, 2008.
From the conclusion. A very small number of
pharmacy-focused journals adhere to the OA paradigm of access. However,
journals that adopt some elements of the OA model, chiefly free
accessibility, may be more likely to be cited than traditional
journals. Pharmacy practitioners, educators, and researchers could
benefit from the advantages that OA offers but should understand its
financial disadvantages.
The same issue has an editorial by C. Richard Talley,
Open-access
publishing: why not? accessible only to subscribers
Suber OAN comment 12 Aug
"
Shafi, Sheikh Mohammad, S.
Research
impact of open access contributions across disciplines
ELPUB2008. Open Scholarship: Authority, Community, and
Sustainability in the Age of Web 2.0 - Proceedings of the 12th
International Conference on Electronic Publishing held in Toronto,
Canada 25-27 June 2008 / Edited by: Leslie Chan and Susanna Mornati,
2008, pp. 343-350
Norris, M., Oppenheim, C., and Rowland, F.
The
citation advantage of open-access articles (full text
requires subscription; abstract only)
Journal of the American Society for
Information Science and
Technology, Vol. 59, No. 12, 2008, 1963-1972, published online:
9 July 2008
Norris et al. full paper
http://elpub.scix.net/cgi-bin/works/Show?_id=335_elpub2008
Added 28 July 2008
Top five most-cited papers from this bibliography, as measured by
Google Scholar
1 Lawrence, S., Free
online availability substantially increases a paper's impact, Nature, 31 May 2001 GS
Biblio
2 Harnad, S. and Brody, T., Comparing
the Impact of Open Access (OA) vs. Non-OA Articles in the Same Journals, D-Lib Magazine, Vol. 10
No. 6, June 2004
GS
Biblio
3 Antelman, K., Do
Open-Access Articles Have a Greater Research Impact? College and Research Libraries,
65(5):372-382, September 2004 GS
Biblio
4 Eysenbach, G., Citation
Advantage of Open Access Articles,
PLoS Biology, Volume 4, Issue 5, May 2006 GS
Biblio
5 Harnad, S., et al., The Access/Impact
Problem and the Green and Gold Roads to Open Access
Serials Review, Vol. 30,
No. 4, 310-314 GS
Biblio
Shorter version: The
green and the gold roads to Open Access
Nature, Web Focus: access to the literature, May 17, 2004 GS
Biblio
Added 28 July 2008 Evans,
J. A. (2008)
Electronic
Publication and the Narrowing of Science and Scholarship (full text
requires subscription; abstract only)
Science, Vol. 321, No. 5887,
18 July 2008, 395 - 399
From the abstract: "Using a database of 34 million articles, their
citations (1945 to 2005), and online availability (1998 to 2005), I
show that as more journal issues came online, the articles referenced
tended to be more recent, fewer journals and articles were cited, and
more of those citations were to fewer journals and articles. The forced
browsing of print archives may have stretched scientists and scholars
to anchor findings deeply into past and present scholarship. Searching
online is more efficient and following hyperlinks quickly puts
researchers in touch with prevailing opinion, but this may accelerate
consensus and narrow the range of findings and ideas built upon."
This work was funded by the National Science Foundation. See the NSF press
release and video interview with James Evans
See also the news feature Great
minds think (too much) alike, The
Economist, July 17th 2008
Comment on this paper:
Davis, P. (2008), The
Paradox of Online Journals,
The Scholarly Kitchen blog, July 18, 2008: "This does not mean that
total citations are dropping, only that fewer articles get cited more.
While it is tempting to eulogize the end of scholarship, this article
may signify that the dissemination of science is working more
efficiently than ever. The institution of science values the progress
of discovery followed by a consensus and closure of debate. That more
of the literature is effectively being ignored may not necessarily
signal a bad thing (although it may concern those who are not read)."
Hooker, Bill (2008), An
Open Access partisan's view,
Open Reading Frame blog, 19 July 2008: "Evans seems to me to gloss over
the question of what proportion of the online archives are freely
available, and what effect that has on the phenomenon he is attempting
to model. ... the driving force in Evans' suggested "narrow[ing of] the
range of findings and ideas built upon" is not online access per se but
in fact commercial access, with its attendant question of who can
afford to read what. Evans' own data indicate that if the online access
in question is free of charge, the apparent narrowing effect is
significantly reduced or even reversed."
For collected comments on this paper, see Electronic
Publication and the Narrowing of Science and Scholarship. Really? A
Blog Around the Clock, July 19, 2008
Suber, P. (2008), Online
researchers have access to more articles but cite fewer, Open
Access News, July 17, 2008: "Evans' results also appear to conflict
with a recent study by Arthur Eger."
Added 28 July 2008 Eger,
A. (2008)
Database
statistics applied to investigate the effects of electronic information
services on publication of academic research – a comparative study
covering Austria, Germany and Switzerland
GMS Medizin - Bibliothek - Information, June 26, 2008
Findings on increased usage of online full text articles leading to
increased publication, but says nothing on the effects of such access
on citation practices
Added 28 July 2008 Adler,
R., Ewing, J. (Chair) and Taylor, P.
Citation
Statistics (pdf 26pp)
Joint Committee on Quantitative Assessment of Research, International
Mathematical Union, IMU-ICIAM-IMS, 6/11/2008
Comment on this paper:
Citation
statistics, American Scientist Open Access
Forum
Armbruster, C. (2008), 12 June: "after reading the
report, I would caution against dismissing it. Science and scientists
should be concerned about the politicisation of metrics. Politicisation
comes from governments and research funders but is also going on inside
academic institutions. Moreover, in a general sense the citation and
usage metrics currently available are not 'fit for purpose'. Worse
still, politicisation carries with it the significant risk of arresting
the development of tools for metric research evaluation. ... All we
have at the moment are some 'quick fix metrics'. And these are
increasingly used to make and legitimate all kinds of decisions. It is
thus welcome that mathematicians and statisticians scrutinise current
practices and show up the lack of validity and reliability of many
measures, technical faults as well as the misguided judgements of
peers, university management, funding agencies and government."
Oppenheim, C. (2008), 12 June: "it fails to address the
fundamental issue, which is: citation and other metrics correlate
superbly with subjective peer review. Both methods have their faults,
but they are clearly measuring the same (or closely related) things.
Ergo, if you have evaluate research in some way, there is no reason NOT
to use them! It also keeps referring to examples from the field
of maths, which is a very strange subject citation-wise."
Harnad, S. (2008), Citation Statistics:
International Mathematical Union Report,
Open Access Archivangelism blog, June 15, 2008: "what all this
valuable, valid cautionary discussion overlooks is not
only the possibility but the empirically demonstrated fact that there
exist metrics that are highly correlated with human expert rankings. It
follows that to the degree that such metrics account for the same
variance, they can substitute for the human rankings. The substitution
is desirable, because expert rankings are extremely costly in terms of
expert time and resources. Moreover, a metric that can be shown to be
highly correlated with an already validated variable predictor variable
(such as expert rankings) thereby itself becomes a validated predictor
variable. And this is why the answer to the basic question of whether
the RAE's decision to convert to metrics was a sound one is: Yes."
Bensman,
S. J. (2008): IMU
Critique of Citation Analysis, SIGMETRICS, 27 June
2008: "I checked the distribution of mathematics journals by impact
factor in the 2007 SCI JCR. It was as I suspected. The range of impact
factors was only from 0.108 to 2.739--extraordinarily low and
tight--and the top journals on the impact factor had no review
articles. This is suggestive of an extremely random citation pattern
with no development of consensual paradigms. Therefore, math acts like
a humanities in terms of its literature use, and citation analysis is
probably not applicable to this discipline. If citation analysis is
used, it has to be backed by other measures."
Added 28
July 2008 Taraborelli, D. (2008)
Soft peer review.
Social software and distributed scientific evaluation (pdf 12pp)
In Proceedings of the 8th
International Conference on the
Design of Cooperative Systems (COOP 08), Carry-Le-Rouet, France,
May
20-23, 2008
From the abstract: "I analyze the contribution that social bookmarking
systems can provide to the problem of usage-based metrics for
scientific evaluation. I suggest that collaboratively aggregated
metadata may help fill the gap between traditional citation-based
criteria and raw usage factors."
Added 28 July 2008 Harnad,
S., Brody, T., Vallières, F., Carr, L., Hitchcock, S., Gingras,
Y., Oppenheim, C., Hajjem, C. and Hilf, E. R. (2008)
The
Access/Impact Problem and the Green and Gold Roads to Open Access: An
Update
Serials Review, Vol. 34, Issue
1, March 2008, 36-40, available online 6 March 2008
also available from ECS EPrints, 06 Jun 2008
http://eprints.ecs.soton.ac.uk/15852/
Update to the paper published in Serials
Review, 30(4), 2004
Added 28 July 2008 Harnad,
S. (2008)
Validating
research performance metrics against peer rankings
Ethics in Science and Environmental
Politics, Vol. 8, No. 1, June 03, 2008, 103-107
Added 28 July 2008 Knowlton,
S. A. (2007)
Continuing
use of print-only information by researchers
J Med Libr Assoc., 95(1):
83–88, January 2007
"to study the question, “Are researchers still accessing and using
material issued only in print?,” a group of journals was selected, and
the impact factor of each was tracked over the period 1993–2003.
Conclusion: the online status of a journal is not sufficient to
override all other
considerations by researchers when they choose which material to cite."
Added 28 July 2008 Jacsó,
P. (2006)
Open
Access to Scholarly Full Text Documents (pdf 8pp)
Online Information Review, 30(5) 2006, 587-594
J
Med Libr Assoc., 93(2): 223–228, April 2005
From the conclusion: "It is possible that electronic access to
information (i.e., online
databases) has had a positive impact on the number of articles faculty
will cite. Results of this study suggest, at this point, that faculty
are still accessing the print-only collection, at least for research
purposes, and are therefore not sacrificing quality for convenience."
Added 28 July 2008 Rossner,
M., Van Epps, H. and Hill, E.
Show me the
data (editorial)
The Journal of Cell Biology,
Vol. 179, No. 6, 1091-1092, published online December 17, 2007
"Just as scientists would not accept the findings in a scientific paper
without seeing the primary data, so should they not rely on Thomson
Scientific's impact factor, which is based on hidden data. As more
publication and citation data become available to the public through
services like PubMed, PubMed Central, and Google Scholar®, we hope
that
people will begin to develop their own metrics for assessing scientific
quality rather than rely on an ill-defined and manifestly unscientific
number."
Comment on this paper:
Pendlebury, D. A., Thomson
Scientific Corrects Inaccuracies In Editorial,
Thomson Reuters Citation Impact Forum, undated: "Rossner, Van Epps, and
Hill (argue) that Thomson Scientific’s impact factor measure for the
evaluation of journals should not be trusted since an article data set
purchased from Thomson by The Rockefeller University Press did not
exactly replicate the Journal Citation Reports data for its own -- and
selected other -- journals. When these data were questioned by The
Rockefeller University Press, Thomson staff explained precisely the
content of the data, as well as its derivation and use. Unfortunately
for the readers of the Rossner editorial, the authors misunderstood
much and as a result, misled readers about several matters, not only
regarding the data but what Thomson representatives did and said from
June to September 2007 in many email exchanges."
Added 28 July 2008 Harzing, A. W.K., van der Wal, R.
(2008) Google
Scholar as a new source for citation analysis, Ethics in Science and Environmental
Politics, Vol. 8, No. 1, June 03, 2008, 61-73
Added 28 July 2008 Jacsó,
P. (2008) The
pros and cons of computing the h-index using Google Scholar. Online Information Review, 32(3)
2008, 437-452
Added 28 July 2008 Jacsó,
P. (2008) Google
Scholar, Online, Mar/Apr
2008, 53-54
Added 28 July 2008 Jacsó,
P. (2008) Live
Search Academic, Gale, Reference Reviews, Péter's Digital
Reference Shelf, April 2008
Added 28 July 2008 Jacsó,
P. (2008) Windows
Live Academic, Online,
Sep/Oct 2006, 59-60
Added 28 July 2008 Jacsó,
P. (2008) Scopus
(2008 Winter Release), Gale, Reference Reviews, Péter's
Digital Reference Shelf, November 2007
Added 28 July 2008 Jacsó,
P. (2008) Web
of Science, Gale, Reference Reviews, Péter's Digital
Reference Shelf, January 2007
Added 28 July 2008 Jacsó,
P. (2008) CiteBase
Search, Online, Sep/Oct
2004, 57-58
Added 28 July 2008 Jacsó,
P. (2008)
The
Plausibility of Computing the H-index of Scholarly Productivity and
Impact Using Reference Enhanced Databases
Online Information Review,
32(2) 2008, 266-283