The Open Citation Project - Reference Linking and Citation Analysis for Open Archives

See also Papers produced by the project

The effect of open access and downloads ('hits') on citation impact: a bibliography of studies

Find you way through the bibliography
Latest additions
Studies with original data
Web tools for measuring impact | Comparative reviews
Background
The financial imperative: correlating research access, impact and assessment | Citation analysis, indexes and impact factors | Open access

OA impact biblio rapid reader
Top five most-cited papers from this bibliography, as measured by Google Scholar

1 Lawrence, S., Free online availability substantially increases a paper's impact, Nature, 31 May 2001 GS Biblio

2 Harnad, S. and Brody, T., Comparing the Impact of Open Access (OA) vs. Non-OA Articles in the Same Journals, D-Lib Magazine, Vol. 10 No. 6, June 2004 GS Biblio

3 Antelman, K., Do Open-Access Articles Have a Greater Research Impact? College and Research Libraries, 65(5):372-382, September 2004 GS Biblio

4 Eysenbach, G., Citation Advantage of Open Access Articles, PLoS Biology, Volume 4, Issue 5, May 2006 GS Biblio

5 Harnad, S., et al., The Access/Impact Problem and the Green and Gold Roads to Open Access Serials Review, Vol. 30, No. 4, 310-314 GS Biblio
Shorter version: The green and the gold roads to Open Access Nature, Web Focus: access to the literature, May 17, 2004 GS Biblio

@ 28 July 2008

Despite significant growth in the number of research papers available through open access, principally through author self-archiving in institutional archives, it is estimated that only c. 20% of the number of papers published annually are open access. It is up to the authors of papers to change this. Why might open access be of benefit to authors? One universally important factor for all authors is impact, typically measured by the number of times a paper is cited (some older studies have estimated monetary returns to authors from article publication via the role citations play in determining salaries). Recent studies have begun to show that open access increases impact. More studies and more substantial investigations are needed to confirm the effect, although a simple example demonstrates the effect.

This chronological bibliography is intended to describe progress in reporting these studies; it also lists the Web tools available to measure impact. It is a focused bibliography, on the relationship between impact and access. It does not attempt to cover citation impact, or other related topics such as open access, more generally, although some key papers in these areas are listed as jump-off points for wider study.

Last updated 10 November 2008; first posted 15 September 2004. Please email additions, corrections or comments to Steve Hitchcock.

This bibliography cited to support the case for the Open Access Motion passed by the Stanford University School of Education

Q&A about the Harvard policy (29 June 2008) https://mx2.arl.org/Lists/SPARC-OAForum/Message/4449.html

"excellent bibliography"

Peter Suber, American Scientist Open Access Forum (28 September 2005) http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/4804.html

"This ongoing chronological bibliography may be worth bookmarking and checking every few months. There’s very little annotation, but it’s a good brief bibliography on a narrow - but important - subject."

Walt Crawford, Cites & Insights (November 2004, p13) http://citesandinsights.info/civ4i13.pdf

Citations of this bibliography found by Google Scholar
Web pages that link to this bibliography found by Google

Latest additions

See also latest papers on impact factors added 10 November 2008.

Added 10 November 2008 Kousha, K. (2008)
Characteristics of Open Access Web Citation Network: A Multidisciplinary Study
Proceedings of WIS 2008, Fourth International Conference on Webometrics, Informetrics and Scientometrics & Ninth COLLNET Meeting (Berlin, 28 July - 1 August 2008), edited by H. Kretschmer and F. Havemann, October 2008

Added 10 November 2008 Lariviere, V., Gingras, Y. and Archambault, E. (2008)
The decline in the concentration of citations, 1900-2007
arXiv.org, arXiv:0809.5250v1 [physics.soc-ph], 30 Sep 2008
From the abstract: "This paper challenges recent research (Evans, 2008) reporting that the concentration of cited scientific literature increases with the online availability of articles and journals. ... contrary to what was reported by Evans, the dispersion of citations is actually increasing."

Comment on this paper:
Davis, P., Citation Controversy, the scholarly kitchen blog, 13 October 2008: "a new analysis taking aim at (Evans') diversity claim. ... Unlike Evan's article, this paper does not require knowledge of negative binomial regression, or any advanced statistics for that matter; and because of the simplicity and descriptive approach to their analysis, it is very convincing. Granted, Evans is using a different approach, looking at the effect of when journals became available online on citation behavior and whether commercial access or free access changes the outcome. For that reason, we should not discount the merit of Evan's paper. ... What makes this controversy interesting is that both studies make theoretical sense. Like many scientific controversies, the argument over citation diversity will move toward consensus and closure. For the meantime, the debate remains open."

Added 10 November 2008 Henneken, E. A., Kurtz, M. J., Accomazzi, A., Grant, C. S., Thompson, D., Bohlen, E. and Murray, S. S. (2008)
Use of Astronomical Literature - A Report on Usage Patterns
arXiv.org, arXiv:0808.0103v1 [cs.DL], 1 Aug 2008

Added 10 November 2008 Dietrich, J. P. (2008)
Disentangling visibility and self-promotion bias in the arXiv: astro-ph positional citation effect
arXiv.org, arXiv:0805.0307v2 [astro-ph], 25 Jun 2008, in Publications of the Astronomical Society of the Pacific, 120 (869): 801-804

Comment on this paper:
Harnad, S., Self-Promotion Bias in Arxiv Deposit Listings, Open Access Archivangelism blog, 5 August 2008: "The authors rightly point out that in a high-output field like astrophysics, visibility is an important factor in usage and citations, and authors need alerting and navigation aids based on importance, relevance and quality, rather than on random timing and author self-promotion biasses. I would add that in fields -- whether high- or low-output -- that, unlike astrophysics, are not yet OA, accessibility itself probably has much the same sort of effect on citations that visibility does in an OA field like astrophysics. (Even maximized visibility cannot make articles accessible to those who cannot afford access to the full-text.)"
Davis, P., The Importance of Being First, the scholarly kitchen blog, 7 August 2008: " an interesting paper indeed, and has its theoretical explanation in attention economics. Physicists, like everyone else, look to the order of one's paper as a heuristic for deciding what to read. This is no different than using other tools, like citations or references as tools to guide a scientists to what is important."
see also Suber, P., Visibility beyond open access, SPARC Open Access Newsletter, issue #87, July 2, 2005

Added 10 November 2008 Oppenheim, C. (2008)
Out with the old and in with the new: The RAE, bibliometrics and the new REF (first page pdf; full text requires subscription)
Journal of Librarianship and Information Science, 40 (3): 147-149, September 2008

Added 25 September 2008 Davis, P. M.
Author-choice open access publishing in the biological and medical literature: a citation analysis
arXiv.org, arXiv:0808.2428v1 [cs.DL], 18 Aug 2008, forthcoming in the Journal of the American Society for Information Science & Technology
This study is a follow-up to the controlled trial of open access publishing published in the BMJ: "According to a study of 11 biological and medical journals that allow authors the choice of making their articles freely available from the publisher's website, few show any evidence of a citation advantage. For those that do, the effect appears to be diminishing over time. ... (the paper) analyzed over eleven thousand articles published in journals since 2003, sixteen hundred of these articles (15%) adopting the author-choice open access model."

Comment on this paper:
Harnad, S., Confirmation Bias and the Open Access Advantage: Some Methodological Suggestions for Davis's Citation Study, Open Access Archivangelism blog, August 25, 2008: "The outcome, confirming previous studies (on both paid and unpaid OA), is a significant OA citation advantage, but a small one (21%, 4% of it correlated with other article variables such as number of authors, references, and pages). The author infers that the size of the OA advantage in this biomedical sample has been shrinking annually from 2004-2007, but the data suggest the opposite." (This blog entry includes a line-by-line commentary on the full paper)

Added 25 September 2008 Clauson, K. A., Veronin, M. A., Khanfar, N. M. and Lou, J. Q.
Open-access publishing for pharmacy-focused journals (full text requires subscription; summary only)
American Journal of Health-System Pharmacy, Vol. 65, No. 16, 1539-1544, August 15, 2008
From the conclusion. A very small number of pharmacy-focused journals adhere to the OA paradigm of access. However, journals that adopt some elements of the OA model, chiefly free accessibility, may be more likely to be cited than traditional journals. Pharmacy practitioners, educators, and researchers could benefit from the advantages that OA offers but should understand its financial disadvantages.
The same issue has an editorial by C. Richard Talley, Open-access publishing: why not? accessible only to subscribers

Comment on this editorial:
Suber, P., OA pharma journals cited more than non-OA pharma journals, Open Access News, 12 August 2008: "(Talley) acknowledges that OA to pharma journals "might indirectly improve public health", but tries to explain why AJHP is not OA. He starts with a potshot at the article by Clauson et al., which he just agreed to publish, by asserting flatly that "Davis et al. found that open-access publishing does not increase article citations." He doesn't attempt to reconcile the Clauson and Davis studies, and doesn't mention that the Davis study found no short-term citation increase while (link to this bibliography) dozens of previous studies have found long-term citation increases."

Added 25 September 2008 Sheikh Mohammad, S.
Research impact of open access contributions across disciplines
12th International Conference on Electronic Publishing (ElPub 2008), Toronto, June 25-27, 2008

Added 25 September 2008 Norris, M., Oppenheim, C., and Rowland, F.
The citation advantage of open-access articles (full text requires subscription; abstract only)
Journal of the American Society for Information Science and Technology, Vol. 59, No. 12, 2008, 1963-1972, published online: 9 July 2008
From the abstract: "Of a sample of 4,633 articles examined, 2,280 (49%) were OA and had a mean citation count of 9.04 whereas the mean for (toll access) TA articles was 5.76. There appears to be a clear citation advantage for those articles that are OA as opposed to those that are TA. This advantage, however, varies between disciplines, with sociology having the highest citation advantage, but the lowest number of OA articles, from the sample taken, and ecology having the highest individual citation count for OA articles, but the smallest citation advantage. Tests of correlation or association between OA status and a number of variables were generally found to weak or inconsistent. The cause of this citation advantage has not been determined."

Simple and practical example

Citation analysis is specialised and difficult. To make the case for, or against, a claim such as 'open access increases impact' requires a lot of the reader, who may not be a specialist but who wants to try and understand the point at issue and decide if it has any relevance to him or her. The following simple example is included for this reason, not as proof but as evidence of the effect within a particular domain. Draw your own conclusions, and then read the more detailed evidence of the bibliography if you are still interested.

"Measuring the effect for physics or astronomy is easy. This link returns the number of articles published in the Astrophysical Journal in 2003 and their number of citations.

"This next link shows the number of these papers which are available OA in the arXiv, and their citations.

"The result is that 75% of the papers are in the arXiv, and they represent 90% of the citations, a 250% OA effect.

"By replacing ApJ with the mnemonic for any other physics or astronomy journal one can repeat the measurement; for Nuclear Physics A (NuPhA) one gets that 32% of the articles are in the arXiv, and they represent 78% of the citations, a 740% OA effect."
From Michael Kurtz, American Scientist Open Access Forum, 28 September 2005 http://users.ecs.soton.ac.uk/harnad/Hypermail/Amsci/4807.html
Note, the database links are 'live', i.e. they return the current database figures, not the exact figures on which Michael Kurtz would have based his calculations, but the percentages quoted are unlikely to change dramatically, in the short term at least.

Elucidation of calculation (by Stevan Harnad, figures valid on 22 July 2007)
For ApJ:
TOT: articles 2592 citations 70732
Arx: articles 1943 citations 62586 c/a 32.21 (rounded to 32)
Non: articles 649 citations 8146 c/a 12.55 (rounded to 13)
Then 32/13 = 2.5 (250%)

For NuPhA:
TOT: articles 1134 citations 4451
Arx: articles 344 citations 3225 c/a 9.375
Non: articles 790 citations 1226 c/a 1.552
Then 9.375/1.553 = 6.041 (600%)

Michael Kurtz comments: "The differences in (NuPhA: 740% to 600% effect) results are because the database has changed over the past two years since I did it. There is a systematic error in the calculations for Nuclear Physics A (Elsevier does not give us the references) so the results will be higher than the true value. Physical Review C (Nuclear Physics) has an OA advantage number of 221%, the systematic in this case is small and in the other direction."

Studies with original data

Highlights

Lawrence (2001) was the first to publish data recognising the trend for online publication to increase impact, confirmed for open access papers by the work of the Open Citation Project based on arXiv (e.g. Harnad and Brody, D-Lib, 2004), and by Kurtz et al. (2004a, 2003a) looking at the NASA Astrophysics Data System. Commenting on Harnad and Brody (D-Lib, 2004) in Open Access News, Peter Suber said:

This is an important article. It's the first major study since the famous Lawrence paper documenting the proposition that OA increases impact. It's also the first to go beyond Lawrence in scope and method in order to answer doubts raised about his thesis. By confirming that OA increases impact, it gives authors the best of reasons to provide OA to their own work (21 June 2004)

Broader collaborations have emerged to extend these findings (e.g. Brody et al. 2004).

Open access has become feasible because of the move towards online publication and dissemination. A new measure that becomes possible with online publication is the number of downloads or 'hits', opening a new line of investigation. Brody et al. have been prominent in showing there is a correlation between higher downloads and higher impact, particularly for high impact papers, holding out the promise not just for higher impact resulting from open access but for the ability to predict high impact papers much earlier, not waiting years for those citations to materialise (e.g. Brody and Harnad 2004, in prep.). The effect can be verified with the Correlation Generator (below).

(Note. The latest listings might include preprints, or even pre-preprints. This area of study is effectively a work in progress, and as such the list is intended to raise awareness of the most recent results, even where these may not be definitive or final versions. Check back for definitive versions.)

Added 4 August 2008 Davis, P.M., Lewenstein, B. V., Simon, D. H., Booth, J. G. and Connolly, M. J. L. (2008)
Open access publishing, article downloads, and citations: randomised controlled trial
BMJ, 2008;337:a568, published 31 July 2008

Comment on this paper:
Anderson, K. (2008), Open Access Doesn't Drive Citations, The Scholarly Kitchen blog, July 31, 2008: "First, a caveat. Phil Davis is a fellow blogger on this site. This study has important advantages over prior studies. It was randomized by the researchers, so authors or publishers didn't select which articles were made open access. It was done prospectively, so that the data were analyzed from point zero forward. These are crucial advantages in study design, in my opinion, and place this study head and shoulders above any prior study asserting a citation advantage. In fact, authors and editors often make studies they think are more significant free or push them online early. Retrospective, non-randomized studies fall prey to all sorts of problems because of these confounding effects."
Harnad, S. (2008), Davis et al's 1-year Study of Self-Selection Bias: No Self-Archiving Control, No OA Effect, No Conclusion, Open Access Archivangelism blog, July 31, 2008: "To show that the OA advantage is an artefact of self-selection bias (or of any other factor), you first have to produce the OA advantage and then show that it is eliminated by eliminating self-selection bias (or any other artefact). This is not what Davis et al. did. They simply showed that they could detect no OA advantage one year after publication in their sample. This is not surprising, since most other studies, some based on hundreds of thousands of articles, don't detect an OA advantage one year after publication either. It is too early. To draw any conclusions at all from such a 1-year study, the authors would have had to do a control condition, in which they managed to find a sufficient number of self-selected, self-archived OA articles (from the same journals, for the same year) that do show the OA advantage, whereas their randomized OA articles do not. In the absence of that control condition, the finding that no OA advantage is detected in the first year for this particular sample of 247 out of 1619 articles in 11 physiological journals is completely uninformative. The authors did find a download advantage within the first year, as other studies have found. This early download advantage for OA articles has also been found to be correlated with a citation advantage 18 months or more later. The authors try to argue that this correlation would not hold in their case, but they give no evidence (because they hurried to publish their study, originally intended to run four years, three years too early.)"
Eysenbach, G. (2008), Word is still out: Publication was premature, BMJ Rapid Responses, 1 August 2008: "While parts of the paper will be welcomed by most Open Access advocates as far as the access/usage data are concerned, showing (unsurprisingly) a significant increase in access and use of Open Access articles compared to non-OA articles, other parts of the paper are more controversial (to be diplomatic). Davis et al failed to show a citation advantage after 9-12 months, from which they conclude that "the citation advantage from open access reported widely in the literature may be an artifact of other causes." Jumping to these conclusions after only 9-12 months is quite irritating ... What surprised (and bothered) me especially is that Davis et al. cites my PNAS data published in PLoS Biology to justify his short observation period. While I indeed found already a statistically significant difference after only 4-10 months (1.2 citations in the nOA group versus 1.5 citations in the OA group), most citations appeared after 10-16 months (4.5 versus 6.4) and 16-22 months (8.9 vs 13.1). In summary, this would have been an important and much more credible paper if it would have been published in 2-3 years as opposed to a salami approach."
More Rapid Responses to this paper

Added 28 July 2008 Evans, J. A. (2008)
Electronic Publication and the Narrowing of Science and Scholarship (full text requires subscription; abstract only)
Science, Vol. 321, No. 5887, 18 July 2008, 395-399
From the abstract: "Using a database of 34 million articles, their citations (1945 to 2005), and online availability (1998 to 2005), I show that as more journal issues came online, the articles referenced tended to be more recent, fewer journals and articles were cited, and more of those citations were to fewer journals and articles. The forced browsing of print archives may have stretched scientists and scholars to anchor findings deeply into past and present scholarship. Searching online is more efficient and following hyperlinks quickly puts researchers in touch with prevailing opinion, but this may accelerate consensus and narrow the range of findings and ideas built upon."
This work was funded by the National Science Foundation. See the NSF press release and video interview with James Evans
See also the news feature Great minds think (too much) alike, The Economist, July 17th 2008

Comment on this paper:
Davis, P. (2008), The Paradox of Online Journals, The Scholarly Kitchen blog, July 18, 2008: "This does not mean that total citations are dropping, only that fewer articles get cited more. While it is tempting to eulogize the end of scholarship, this article may signify that the dissemination of science is working more efficiently than ever. The institution of science values the progress of discovery followed by a consensus and closure of debate. That more of the literature is effectively being ignored may not necessarily signal a bad thing (although it may concern those who are not read)."
Hooker, Bill (2008), An Open Access partisan's view, Open Reading Frame blog, 19 July 2008: "Evans seems to me to gloss over the question of what proportion of the online archives are freely available, and what effect that has on the phenomenon he is attempting to model. ... the driving force in Evans' suggested "narrow[ing of] the range of findings and ideas built upon" is not online access per se but in fact commercial access, with its attendant question of who can afford to read what. Evans' own data indicate that if the online access in question is free of charge, the apparent narrowing effect is significantly reduced or even reversed."
For collected comments on this paper, see Electronic Publication and the Narrowing of Science and Scholarship. Really? A Blog Around the Clock, July 19, 2008
Suber, P. (2008), Online researchers have access to more articles but cite fewer, Open Access News, July 17, 2008: "Evans' results also appear to conflict with a recent study by Arthur Eger."
Harnad, S. (2008), Are Online and Free Online Access Broadening or Narrowing Research? Open Access Archivangelism blog, August 4, 2008: "In one of the few fields where [the narrowing citations effect found by Evans] can be and has been analyzed thoroughly, astrophysics, which effectively has 100% Open Access (OA) (free online access) already, Michael Kurtz too found that with free online access to everything, reference lists became (a little) shorter, not longer, i.e., people are citing (somewhat) fewer papers, not more, when everything is accessible to them. ... Are online and free online access broadening or narrowing research? They are broadening it by making all of it accessible to all researchers, focusing it on the best rather than merely the accessible, and accelerating it."

Added 28 July 2008 Eger, A. (2008)
Database statistics applied to investigate the effects of electronic information services on publication of academic research - a comparative study covering Austria, Germany and Switzerland
GMS Medizin - Bibliothek - Information, June 26, 2008
Findings on increased usage of online full text articles leading to increased publication, but says nothing on the effects of such access on citation practices

Updated 25 September 2008 Norris, M., Oppenheim, C. and Rowland, F. (2008)
Open Access Citation Rates and Developing Countries
12th International Conference on Electronic Publishing (ElPub 2008), Toronto, June 25-27, 2008
"the admittedly small number of citations from authors in developing countries do indeed seem to show a higher proportion of citations being given to OA articles than is the case for citations from developed countries."

Added 28 July 2008 Taraborelli, D. (2008)
Soft peer review. Social software and distributed scientific evaluation (pdf 12pp)
In Proceedings of the 8th International Conference on the Design of Cooperative Systems (COOP 08), Carry-Le-Rouet, France, May 20-23, 2008
From the abstract: "I analyze the contribution that social bookmarking systems can provide to the problem of usage-based metrics for scientific evaluation. I suggest that collaboratively aggregated metadata may help fill the gap between traditional citation-based criteria and raw usage factors."

Added 26 May 2008 Cheng, W. H. and Ren, S. L. (2008)
Evolution of open access publishing in Chinese scientific journals (full text requires subscription; abstract only)
Learned Publishing, Vol. 21, No. 2, April 2008, 140-152
From the abstract: "Citation indicators of OA journals were found to be higher than those of non-OA journals."

Added 28 July 2008 Harnad, S., Brody, T., Vallières, F., Carr, L., Hitchcock, S., Gingras, Y., Oppenheim, C., Hajjem, C. and Hilf, E. R. (2008)
The Access/Impact Problem and the Green and Gold Roads to Open Access: An Update
Serials Review, Vol. 34, Issue 1, March 2008, 36-40, available online 6 March 2008
also available from ECS EPrints, 06 Jun 2008 http://eprints.ecs.soton.ac.uk/15852/
Update to the paper published in Serials Review, 30(4), 2004

Added 26 May 2008 Lokker, C., McKibbon, K. A., McKinlay, R.J., Wilczynski, N. L. and Haynes, R. B. (2008)
Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study
BMJ, 2008;336:655-657 (22 March), published 21 February 2008
"Conclusion: Citation counts can be reliably predicted at two years using data within three weeks of publication."

Comment on this paper:
Harnad, S. (2008), Predicting later citation counts from very early data, American Scientist Open Access Forum, 21 April 2008: "This finding reinforces the importance of taking into account as many predictor metrics as possible, though a number of the metrics do seem specific to clinical medical articles. ... We might perhaps make a distinction between static and dynamic metrics. This study was based largely on static metrics, in that they are fixed as of the day of publication. Dynamic metrics like early downloads (which have also been found to predict later citations) were not included, nor were early citation growth metics (also predictive of later citations). ... To my mind, the article reinforces the importance of validating all these metrics, not just against one another, but against peer evaluations, in all fields, as in the RAE 2008 database"

Added 11 February 2008 Chu, H. and Krichel, T. (2008)
Downloads vs. Citations: Relationships, Contributing Factors and Beyond
E-LIS, 9 February 2008, in 11th Annual Meeting of the International Society for Scientometrics and Informetrics, Madrid, 25-27 June 2007
From the abstract: "In a nutshell, an infrastructure that encourages downloading at digital libraries would eventually lead to higher usage of their resources."

Added 26 May 2008 Turk, N. (2008)
Citation impact of Open Access journals (full text requires subscription; summary only)
New Library World, Vol. 109, No. 1/2, January/February 2008, 65-74
Review of the main research about citation impact of Open Access journals, focused on LIS journals.

Added 26 May 2008 Hardisty, D. J. and Haaga, D. A. F. (2008)
Diffusion of Treatment Research: Does Open Access Matter? (pdf 39pp)
Center for the Decision Sciences, Columbia University, in Journal of Clinical Psychology, Vol. 64(7), 1-19 (2008)
From the abstract: "In a pair of studies, mental health professionals were given either no citation, a normal citation, a linked citation, or a free access citation and were asked to find and read the cited article. After one week, participants read a vignette on the same topic as the article and gave recommendations for an intervention. In both studies, those given the free access citation were more likely to read the article, yet only in one study did free access increase the likelihood of making intervention recommendations consistent with the article."

Added 11 February 2008 Kousha, K. and Thelwall, M. (2007)
The Web impact of open access social science research (full-text requires subscription; otherwise abstract only)
Library & Information Science Research, Volume 29, Issue 4, December 2007, 495-507, available online 15 October 2007
preprint http://www.scit.wlv.ac.uk/~cm1993/papers/OpenAccessSocialSciencePreprint.doc (.doc 12pp)
From the abstract: "The results suggest that new types of citation information and informal scholarly indictors could be extracted from the Web for the social sciences."

Added 17 December 2007 Dietrich, J. P. (2007)
The Importance of Being First: Position Dependent Citation Rates on arXiv:astro-ph
arXiv.org, arXiv:0712.1037v1 [astro-ph], 6 December 2007, in Publications of the Astronomical Society of the Pacific, 120 (864): 224-228, February 2008
From the abstract: "We study the dependence of citation counts of e-prints published on the arXiv:astro-ph server on their position in the daily astro-ph listing. ... cannot exclude that increased visibility at the top of the daily listings contributes to higher citation counts as well."

Comment on this paper:
Kurtz, M. J. (2007), Systematics in Citation Statistics: Implications for the "OA Advantage", American Scientist Open Access Forum, 14 December 2007: "shows that systematic increases in citation rates on the order of the 2 to 1 "OA advantage" can be obtained by other, systematic means, such as author bias, as pointed out by myself, Moed and others. Note that OA plays no role in the Dietrich paper, all the articles are from arXiv. "

Added 13 September 2007 Kurtz, M. J. and Henneken, E. A. (2007)
Open Access does not increase citations for research articles from The Astrophysical Journal
arXiv.org, arXiv:0709.0896v1 [cs.DL], 6 September 2007
Abstract: We demonstrate conclusively that there is no "Open Access Advantage" for papers from the Astrophysical Journal. The two to one citation advantage enjoyed by papers deposited in the arXiv e-print server is due entirely to the nature and timing of the deposited papers. This may have implications for other disciplines.

Comment on this paper:
Harnad, S. (2007), Where There's No Access Problem There's No Open Access Advantage, Open Access Archivangelism blog, September 7, 2007: "K & H suggest: "[T]here is no 'Open Access Advantage' for papers from the Astrophysical Journal" because "in a well funded field like astrophysics essentially everyone who is in a position to write research articles has full access to the literature." This seems like a perfectly reasonable explanation for K&H's findings. Where there is no access problem, OA cannot be the cause of whatever higher citation count is observed for self-archived articles. K&H conclude that "[t]his may have implications for other disciplines." It should be evident, however, that the degree to which this has implications for other disciplines depends largely on the degree to which it is true in other disciplines that "essentially everyone who is in a position to write research articles has full access to the literature."

Added 22 August 2007 Sotudeh, H. and Horri, A. (2007)
The citation performance of open access journals: A disciplinary investigation of citation distribution models (full-text subscribers only; no abstract)
Journal of the American Society for Information Science and Technology, Vol. 58, No. 13, 2007, 2145-2156, published online August 17, 2007
From the conclusion: "To sum up, the similarity of the science system across OAJ and NOAJ boundaries has been confirmed. We see this as further evidence of OA's widespread recognition by scientific communities. However, because the magnitudes of the exponents found in this study are lower than what was previously observed for the whole system, OA may currently perform at a slightly lower level. According to the models used in this study, the citation distributions between fields are strongly disproportionate in Life Sciences and Engineering and Material Sciences, favoring larger fields in the former, but smaller fields in the latter. However, the distributions tend to be rather linear in the Natural Sciences."

Added 13 September 2007 Brody, T., Carr, L., Gingras, Y., Hajjem, C., Harnad, S. and Swan, A. (2007)
Incentivizing the Open Access Research Web: Publication-Archiving, Data-Archiving and Scientometrics
CTWatch Quarterly, Vol. 3, No. 3, August 2007

Added 22 August 2007 Lin, S.-K. (2007)
Editorial: Non-Open Access and Its Adverse Impact on Molecules
Molecules, 12, 1436-1437, 16 July 2007
The point of this short editorial is clear, that the difference between the OA and non-OA content in the journal Molecules is clearly reflected in higher citations for the former. The context could be clearer, however. The OA/non-OA history of the journal, especially prior to the period under review (2005-6), is not elaborated and familiarity with the journal is assumed.

Added 22 August 2007 Taylor, D. (2007)
Looking for a Link: Comparing Faculty Citations Pre and Post Big Deals
Electronic Journal of Academic and Special Librarianship, v.8 no.1 (Spring 2007)
Note. The Big Deal is where a library or consortium of libraries subscribes to a larger package of a publisher's journals than they would have if they had subscribed to journals individually. Big Deals are claimed to improve access for an institution's users. "Pre Big Deal, the percentage of citations to journals that are part of Big Deals but were previously not subscribed to was an average of 2.6%. Post Big Deal this increased to an average of 6.1%." There is no analysis or comment on how this result might be affected if it was considering open access.

Added 22 May 2007 Craig, I. D., Plume, A. M., McVeigh, M. E., Pringle, J. and Amin, M. (2007)
Do Open Access Articles Have Greater Citation Impact? A critical review of the literature
Publishing Research Consortium, undated (announced 17 May 2007), Journal of Informetrics, 1 (3): 239-248, July 2007

Comment on this paper:
Suber, P. (2007), Publishers doubt the OA impact advantage, Open Access News, 18 May 2007: "we shouldn't be surprised to see that good relevant literature that is easier to find and retrieve is cited more often than good relevant literature that is harder to find and retrieve. Or, if a careful study concluded that this view is false, then one might expect it to be more careful in summarizing the reasons why."
Harnad, S. (2007), Craig et al.'s review of the OA citation advantage, Open Access Archivangelism blog, 26 May 2007: "I also agree that not one of the studies done so far is without some methodological flaw that could be corrected. But it is also highly probable that the results of the methodologically flawless versions of all those studies will be much the same as the results of the current studies. That's what happens when you have a robust major effect, detected by virtually every study, and only ad hoc methodological cavils and special pleading to rebut each of them with. But I am sure those methodological flaws will not be corrected by these authors, because -- OJ Simpson's "Dream Team" of Defense Attorneys comes to mind -- Craig et al's only interest is evidently in finding flaws and alternative explanations, not in finding out the truth -- if it goes against their client's interests...
Iain D.Craig: Wiley-Blackwell
Andrew M.Plume, Mayur Amin: Elsevier
Marie E.McVeigh, James Pringle: Thomson Scientific"

Added 10 May 2007 Tonta, Y., Ünal, Y. and Al, U. (2007)
The Research Impact of Open Access Journal Articles
E-LIS, 30 April 2007, also in Proceedings ELPUB 2007, the 11th International Conference on Electronic Publishing, Vienna, 13-15 June 2007

Comment on this paper:
Harnad, S. (2007), OA citation impact study: No conclusions possible, American Scientist Open Access Forum, 1 May 2007: "No comparison was made with non-OA journals in the same fields. Hence it is impossible to say whether any of these differences have anything to do with OA. Fields no doubt differ in their average number of citations. Journals no doubt differ too, in subject matter, quality, and citation impct. And it is not clear whether the OA journals in each field are the top, medium or bottom journals, relative to the non-OA journals. No conclusions at all can be drawn from this study. The authors are encouraged to do the necessary controls."

Added 26 May 2008 Sharma, H. P. (2007)
Download plus citation counts - a useful indicator to measure research impact (correspondence, pdf 1pp)
Current Science, 92 (7): 873-873, April 10, 2007

Added 10 May 2007 Piwowar, H. A., Day, R. S. and Fridsma, D. B. (2007)
Sharing Detailed Research Data Is Associated with Increased Citation Rate
PLoS ONE, March 21, 2007
Principal Findings: "We examined the citation history of 85 cancer microarray clinical trial publications with respect to the availability of their data. The 48% of trials with publicly available microarray data received 85% of the aggregate citations. Publicly available data was significantly (p = 0.006) associated with a 69% increase in citations, independently of journal impact factor, date of publication, and author country of origin using linear regression."

Comment on this paper:
Suber, P. (2007), Open data and citation impact, Open Access News, 28 March 2007: " Many studies have shown a correlation between OA articles and citation impact. I believe this is the first study to document a similar correlation between OA data and citation impact. Spread the word to colleagues who are still hoarding data, waiting too long before releasing it, or unable to see any gain for themselves in data sharing."

Added 8 March 2007 Bergstrom, T. C. and Lavaty, R. (2007)
How often do economists self-archive?
eScholarship Repository, University of California, February 8, 2007

Comment on this paper:
Harnad, S. (2007), How often do economists self-archive? (fwd), American Scientist Open Access Forum, 9 February 2007: "An important paper"

Added 26 May 2008 Chapman, S., Nguyen, T. N. and White, C. (2007)
Press-released papers are more downloaded and cited (full text requires subscription; extract only)
Tobacco Control, 16 (1): 71-71, February 2007

Added 22 January 2007 Harnad, S. and Hajjem, C. (2007)
The Open Access Citation Advantage: Quality Advantage Or Quality Bias?
Author blog, Open Access Archivangelism, 21 January 2007
Does the OA Advantage (OAA) occur because authors are more likely to self-selectively self-archive articles that are more likely to be cited (self-selection "Quality Bias": QB), or because articles that are self-archived are more likely to be cited ("Quality Advantage": QA)? Preliminary evidence based on over 100,000 articles from multiple fields, comparing self-selected self-archiving with mandated self-archiving to estimate the contributions of QB and QA to the OAA shows: "Both factors contribute, and the contribution of QA is greater." Includes comment on Moed, H. (2006), The effect of 'Open Access' upon citation impact: An analysis of ArXiv's Condensed Matter Section.

Comment on this paper:
Laloe, F. and Harnad, S., Re: Quality Bias vs Quality Advantage, SIGMETRICS, see installments of this discussion on 20, 21 and 22 February 2007

Added 17 January 2007 Harnad, S. (2007)
Citation Advantage For OA Self-Archiving Is Independent of Journal Impact Factor, Article Age, and Number of Co-Authors
Author blog, Open Access Archivangelism, 17 January 2007
Further comment on Eysenbach, G. (2006), Citation Advantage of Open Access Articles: "The OA-self-archiving advantage remains a robust, independent factor."

Added 17 January 2007 Brody, T. (2007)
Evaluating Research Impact through Open Access to Scholarly Communication
PhD, Electronics and Computer Science, University of Southampton, May 2006, in ECS EPrints, 14 January 2007

Added 8 March 2007 McDonald, J. D. (2007)
Understanding Online Journal Usage: A Statistical Analysis of Citation and Use
Journal of the American Society for Information Science & Technology, 58(1): 39-50, January 1, 2007, also in Caltech Library System Papers and Publications, 18 May 2006

Added 28 July 2008 Knowlton, S. A. (2007)
Continuing use of print-only information by researchers
J Med Libr Assoc., 95(1): 83-88, January 2007
"to study the question, "Are researchers still accessing and using material issued only in print?," a group of journals was selected, and the impact factor of each was tracked over the period 1993-2003.
Conclusion: the online status of a journal is not sufficient to override all other considerations by researchers when they choose which material to cite."

Added 26 May 2008 Walters, G. D. (2006)
Predicting subsequent citations to articles published in twelve crime-psychology journals: Author impact versus journal impact (full text requires subscription; abstract only)
Scientometrics, 69 (3): 499-510, December 2006
"These results suggest that author impact may be a more powerful predictor of citations received by a journal article than the periodical in which the article appears."

Added 23 November 2006 Harnad, S. (2006)
The Self-Archiving Impact Advantage: Quality Advantage or Quality Bias?
Author blog, Open Access Archivangelism, 20 November 2006

Added 19 November 2006 Moed, H. F. (2006)
The effect of 'Open Access' upon citation impact: An analysis of ArXiv's Condensed Matter Section
ArXiv, Computer Science, cs.DL/0611060, 14 November 2006, in Journal of the American Society for Information Science and Technology, Vol. 58, No. 13, 2007, 2145-2156, published online August 30, 2007 http://dx.doi.org/10.1002/asi.20663 (subscriber access only to full text)
"This article statistically analyses how the citation impact of articles deposited in the Condensed Matter section of the preprint server ArXiv, and subsequently published in a scientific journal, compares to that of articles in the same journal that were not deposited in that archive. Its principal aim is to further illustrate and roughly estimate the effect of two factors, 'early view' and 'quality bias', upon differences in citation impact between these two sets of papers ... The analysis provided evidence of a strong quality bias and early view effect. Correcting for these effects, there is in a sample of 6 condensed matter physics journals studied in detail, no sign of a general 'open access advantage' of papers deposited in ArXiv. The study does provide evidence that ArXiv accelerates citation, due to the fact that that ArXiv makes papers earlier available rather than that it makes papers freely available."

Comment on this paper:
Harnad, S. (2007), The Open Access Citation Advantage: Quality Advantage Or Quality Bias?, Open Access Archivangelism, 21 January 2007
Moed, H. (2006) reply on OA to sigmetrics, SIGMETRICS list server, 8 December 2006: On Quality bias. (Harnad) wrote: "The fact that highly-cited articles (Kurtz) and articles by highly-cited authors (Moed) are more likely to be Arxived certainly does not settle the question of cause and effect: It is just as likely that better articles benefit more from Arxiving (QA) as that better authors/articles tend to Arxive/be-Arxived more (QB)". Citation rates may be influenced both by the ‘quality’ of the papers and by the access modality (deposited versus non-deposited). This is why I estimated author prominence on the basis of the citation impact of their non-archived articles only. But even then I found evidence that prominent, influential authors (in the above sense) are overrepresented in papers deposited in ArXiv. But I did more that that. I calculated Arxiv Citation Impact Differentials (CID) at the level of individual authors. Next, I calculated the median CID over authors publishing in a journal. How then do you explain my empirical finding that for some authors the CID is positive, for others it is negative, while the median CID over authors does not significantly differ from zero (according to a Sign test) for all journals studied in detail except Physical Review B, for which it is only 5 per cent? If there is a genuine ‘OA advantage’ at stake, why then does it for instance not lead to a significantly positive median CID over authors? Therefore, my conclusion is that, controlling for quality bias and early view effect, in the sample of 6 journals analysed in detail in my study, there is no sign of a general ‘open access advantage’ of papers deposited in ArXiv’s Condensed Matter Section. ... I hope that more case studies will be carried out in the near future, applying the methodologies I proposed in my paper.
Harnad, S. (2006) The Self-Archiving Impact Advantage: Quality Advantage or Quality Bias?, Open Access Archivangelism, 20 November 2006

Added 13 September 2007 Bollen, J. and Van de Sompel, H. (2006)
Usage Impact Factor: the effects of sample characteristics on usage-based impact metrics
arXiv.org > cs > arXiv:cs/0610154v2 [cs.DL], 26 October 2006, in Journal of the American Society for Information Science and Technology, 59 (1): 136-149, January 1, 2008

Updated 23 November 2006 Mayr, P. (2006)
Constructing experimental indicators for Open Access documents
E-LIS, 05 October 2006, to appear in Research Evaluation, special issue on 'Web indicators for Innovation Systems', 14, 2 (2006)
Author preprint, http://www.ib.hu-berlin.de/~mayr/arbeiten/mayr_RE06.pdf (pdf 9pp)

Added 19 November 2006 Henneken, E. A., Kurtz, M. J., Warner, S., Ginsparg, P., Eichhorn, G., Accomazzi, A., Grant, C. S., Thompson, D., Bohlen, E. and Murray, S. S. (2006)
E-prints and Journal Articles in Astronomy: a Productive Co-existence
ArXiv, Computer Science, cs.DL/0609126, 22 September 2006, in Learned Publishing, Vol. 20, No. 1, January 2007, 16-22

Comment and discussion:
Harnad, S. (2006) The Self-Archiving Impact Advantage: Quality Advantage or Quality Bias?, Open Access Archivangelism, 20 November 2006
Harnad, S. (2006) The Special Case of Astronomy, Open Access Archivangelism, October 14. 2006

Added 28 July 2008 Jacsó, P. (2006)
Open Access to Scholarly Full Text Documents (pdf 8pp)
Online Information Review, 30(5) 2006, 587-594

Added 19 November 2006 Zhang, Y. (2006)
The Effect of Open Access on Citation Impact: A Comparison Study Based on Web Citation Analysis (abstract only)
Libri, September 2006 (Full text for subscribers)

Added 03 August 2006 Metcalfe, T. S. (2006)
The Citation Impact of Digital Preprint Archives for Solar Physics Papers
Solar Physics, Vol. 239, No. 1-2, December 2006, pp. 549-553
also in ArXiv, Astrophysics, astro-ph/0607079, 5 July 2006 http://arxiv.org/abs/astro-ph/0607079
"Most astronomers now use the arXiv.org server (astro-ph) to distribute preprints, but the solar physics community has an independent archive hosted at Montana State University. For several samples of solar physics papers published in 2003, I quantify the boost in citation rates for preprints posted to each of these servers. I show that papers on the MSU archive typically have citation rates 1.7 times higher than the average of similar papers that are not posted as preprints, while those posted to astro-ph get 2.6 times the average. A comparable boost is found for papers published in conference proceedings, suggesting that the higher citation rates are not the result of self-selection of above-average papers."

Added 03 August 2006 Henneken, E. A., Kurtz, M. J., Eichhorn, G., Accomazzi, A., Grant, C., Thompson, D., and Murray, S. S. (2006)
Effect of E-printing on Citation Rates in Astronomy and Physics
Journal of Electronic Publishing, Vol. 9, No. 2, Summer 2006, also in ArXiv, Computer Science, cs.DL/0604061, v2, 5 June 2006 http://arxiv.org/abs/cs/0604061
"It has been observed that papers that initially appear as arXiv e-prints get cited more than papers that do not. Using the citation statistics from the NASA-Smithsonian Astrophysics Data System, we confirm the findings from other studies, we examine the average citation rate to e-printed papers in the Astrophysical Journal, and we show that for a number of major astronomy and physics journals the most important papers are submitted to the arXiv e-print repository first.

Added 03 August 2006 Kousha, K. and Thelwall, M. (2006)
Google Scholar Citations and Google Web/URL Citations: A Multi-Discipline Exploratory Analysis
E-LIS, 05 June 2006, also in Proceedings International Workshop on Webometrics, Informetrics and Scientometrics & Seventh COLLNET Meeting, Nancy (France), May 2006
"we built a sample of 1,650 articles from 108 Open Access (OA) journals published in 2001 in four science and four social science disciplines. We recorded the number of citations to the sample articles using several methods based upon the ISI Web of Science, Google Scholar and the Google search engine (Web/URL citations). For each discipline, we found significant correlations between ISI citations and both Google Scholar and Google Web/URL citations; with similar results when using total or average citations, and when comparing within and across (most) journals."

Added 16 May 2006 Eysenbach, G. (2006)
Citation Advantage of Open Access Articles
PLoS Biology, Volume 4, Issue 5, May 2006
Further evidence for the OA citation advantage, although quite critical of other studies with which its findings broadly agree. This example is based on a small, single journal sample (PNAS: Proceedings of the National Academy of Sciences). Since PNAS offers authors the choice of paying to provide open access to published papers and/or freely self-archiving, a 'Secondary analysis' considers the relative impact of each type of OA, although the number of papers involved is really too small to give this result the weight of the broader findings. The paper is accompanied by two editorials, one in the publishing journal, the other a self-published editorial by the author:
MacCallum, C. J. and Parthasarathy, H. (2006) Editorial: Citation Advantage of Open Access Articles, PLoS Biology, Volume 4, Issue 5, May 2006
Eysenbach, G. (2006) The Open Access Advantage, Journal of Medical Internet Research, 2006;8(2):e8

Comment on this paper:
Harnad, S. on Chronicle of Higher Education News Blog on this paper, 15 May 2006: "The Eysenbach study is certainly not “the first to compare open-access and non-open-access papers from the same journal”. See [this bibliography]"
Harnad, S. eLetter, PLoS, Pipe-Dreams and Peccadillos, on the PLoS editorial, 16 May 2006: "There can be disagreement about what evidence one counts as "solid," but there can be little dispute that prior evidence derived from substantially larger and broader-based samples showing substantially the same outcome can hardly be described as "surprisingly hard to find". The only new knowledge from this small, journal-specific sample was (1) the welcome finding of how early the OA advantage can manifest itself, plus (2) some less clear findings about differences between first- and last-author OA practices, plus (3) a controversial finding that will most definitely need to be replicated on far larger samples in order to be credible: "The analysis revealed that self-archived articles are also cited less often than OA articles from the same journal.""
Author's response eLetter, PLoS, 17 May 2006: "None of the previous papers in [this bibliography] employed a similar methodology, working with data from a "gold-OA" journal. ... Regarding "larger samples" I think rigor and quality (leading to internal validity) is more important than quantity (or sample size). Going through the laborious effort to extract article and author characteristics for a limited number of articles (n=1492) in order to control for these confounders provides scientifically stronger evidence than doing a crude, unadjusted analysis of a huge number of online accessible vs non-online accessible articles, leaving open many alternative explanations. ... contrary to what Harnad said, this study is NOT at all "showing substantially the same outcome". On the contrary, the effect of green-OA -- once controlled for confounders -- was much less than what others have claimed in previous papers."
Harnad, S. blog, Confirming the Within-Journal OA Impact Advantage, 18 May 2006: "with the large, consistent within-journal OA/NOA differences found across all journals, all disciplines and all years in samples four orders of magnitude larger than Eysenbach's, it is not at all clear that controls for those "multiple confounders" are necessary in order to demonstrate the reality, magnitude and universality of the OA advantage. That does not mean the controls are not useful, just that they are not yet telling us much that we don't already know. ... the true measure of the SOA (Self-Archived OA) advantage today (at its 15% spontaneous baseline) is surely not to be found in PNAS but in the statistically far more numerous, hence far more representative full-spectrum of journals that do not yet offer POA (Payed OA). (I would be delighted if those journals took the Eysenbach findings as a reason for offering a POA option! But not at the expense of authors drawing the absurd conclusion -- not at all entailed by Eysenbach's PNAS-specific results -- that in the journals they currently publish in, SOA alone would not confer citation advantages at least as big as the ones we have been reporting.)"
Author's response WebCite blog, The OA debate between an “archivangelist” and an OA researcher, 24 May 2006: "“Confounded” associations between two variables which falsely suggests causality can ONLY be ruled out if one controls for the confounder no matter how “strong, consistent” the effect appears ... Our “events” (citations) are determined by many different “causes”, of which “access” is only one variable – many other variables, including confounders, have to be taken into account.
Harnad, S. blog, The Epidemiology of OA, 26 May 2006: considers each of Eysenbach’s list of confounders, which ones might have an effect and which ones will be tested. "Stay tuned..."
Suber, P. SPARC Open Access Newsletter, issue #98, Gunther Eysenbach confirms the OA impact advantage, 2 June 2006: "There's some controversy about whether some earlier results, especially by Tim Brody, Chawki Hajjem, and Stevan Harnad, are the same or only similar to some of Eysenbach's results. But no one doubts that Eysenbach has new and valid results, or that he has persuasively advanced the case that OA helps authors and journals build their citation impact. While there have been many previous studies of the OA impact advantage, none has made the splash that Eysenbach's has."
Davis, P. Citation advantage of Open Access articles likely explained by quality differential and media effects, PLoS Biology, Responses, 16 January 2007: "The fact that OA articles were more likely to be featured on the front cover of PNAS and covered by the media suggests that other causal explanations may explain the OA advantage. Open Access may be a result – not a cause of – a quality differential which is amplified by the media. While Eysenbach's attempt to control other explanatory variables was excellent, what is needed are true randomized controlled studies of OA publishing."
Harnad, S. blog, Citation Advantage For OA Self-Archiving Is Independent of Journal Impact Factor, Article Age, and Number of Co-Authors, 17 January 2007: "Chawki Hajjem has now done a multiple regression analysis jointly testing (1) article age, (2) journal impact factor, (3) number of authors, and (4) OA self-archiving as separate factors for 442,750 articles in 576 (biomedical) journals across 11 years, and has shown that each of the four factors contributes an independent, statistically significant increment to the citation counts. The OA-self-archiving advantage remains a robust, independent factor.

Added 15 March 2006 Davis, P. M. and Fromerth, M. J. (2006)
Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles? (pdf 12pp)
draft manuscript, ArXiv.org, cs.DL/0603056, 14 March 2006, Scientometics, Vol. 71, No. 2. (May 2007)

Comment on this paper:
Authors Self-Selection bias "While our study confirms the same citation advantage reported by others, it does not attribute Open Access as the cause of more citations, but to Self-Selection. Open Access therefore may be a result, not a cause, of authors promoting higher-quality work."
Liblicense, 14 March 2006
Harnad, S. OAA a causal factor "I think your results are very interesting, but I don't think they have shown that the OA citation advantage (OAA) is all or mostly a self-selection Quality Bias (QB) correlate, rather than being causal. It is still quite plausible that the OAA is a genuine causal factor, but that it has a bigger effect on the high quality/citation end."
SIGMETRICS, 14 March 2006
Qualifying statements: Authors "I am however, troubled by individuals who make universal and unqualified statements like, "Open Access increases citations by 50-250%!"" Harnad, S. "I am such an individual, and I hold by that statement" Authors "The more precise answer is much more subtle, but I understand that a statement like, "open access may provide some citation benefit, but only for prestigious authors who publish in prestigious journals and whose article is already highly-cited", doesn't sound as convincing to administrators and policy makers." Harnad, S. "I don't think it is a correct statement of the thrust of the current body of findings on the OAA. It is merely your interpretation of the result of your own study in 4 maths journals!"
SIGMETRICS, 15 March 2006
Causation or association?: Antelman, K. "Data I collected for philosophy, political science, engineering and mathematics do not support this hypothesis that OA causes more citations for better articles only ... at that time I had not looked at the distribution of OA and non-OA articles by citations. Graphs of those results are posted at http://www.lib.ncsu.edu/staff/kantelman/OA_by_citations.xls. These data show OA citation advantage across all articles with more than zero citations. Authors "The data that Kristin illustrates do not show causation, only association." Harnad, S. The data Phil illustrates likewise do not show causation, only association. ... More fine-tuned causal tests are needed to decide.
Liblicense, from 20 March 2006

Added 16 March 2006 Harnad, S. (2006)
OA Impact Advantage = EA + (AA) + (QB) + QA + (CA) + UA
Author eprint, 14 March 2006, ECS EPrints repository, School of Electronics and Computer Science, University of Southampton

Added 03 August 2006 Mueller, P. S., Murali, N. S., Cha, S. S., Erwin, P. J. and Ghosh, A. K. (2006)
The effect of online status on the impact factors of general internal medicine journals
Netherlands Journal of Medicine, 64 (2): 39-44, February 2006
"becoming available online as FUTON (full text on the Net) is associated with a significant increase in journal impact factor."

Added 30 December 2005 Hajjem, C., Harnad, S. and Gingras, Y. (2005)
Ten-Year Cross-Disciplinary Comparison of the Growth of Open Access and How it Increases Research Citation Impact (pdf 8pp)
IEEE Data Engineering Bulletin, Vol. 28 No. 4, December 2005
also Author eprint, 16 December 2005 http://eprints.ecs.soton.ac.uk/11688/
"In 2001, Lawrence found that articles in computer science that were openly accessible (OA) on the Web were cited substantially more than those that were not. We have since replicated this effect in physics. To further test its cross-disciplinary generality, we used 1,307,038 articles published across 12 years (1992-2003) in 10 disciplines (Biology, Psychology, Sociology, Health, Political Science, Economics, Education, Law, Business, Management). The overall percentage of OA (relative to total OA + NOA) articles varies from 5%-16% (depending on discipline, year and country) and is slowly climbing annually. Comparing OA and NOA articles in the same journal/year, OA articles have consistently more citations, the advantage varying from 25%-250% by discipline and year."

Added 30 December 2005 Hajjem, C., Gingras, Y., Brody, T., Carr, L. and Harnad, S. (2005)
Open Access to Research Increases Citation Impact (.doc 12pp)
Author eprint, 16 December 2005, Technical Report, Institut des sciences cognitives, Université du Québec à Montréal

Added 30 December 2005 Sahu, D.K., Gogtay, N.J. and Bavdekar, S.B. (2005)
Effect of open access on citation rates for a small biomedical journal
Author eprint, December 1, 2005, in Fifth International Congress on Peer Review and Biomedical Publication, Chicago, September 16-18, 2005
"We assessed the influence of OA on citations rates for a small, multi-disciplinary journal which adopted OA without article submission or article access fee. DESIGN The full text of articles published since 1990 were made available online in 2001. Citations for these articles as retrieved using Web of Science, SCOPUS, and Google Scholar were divided into two groups - the pre-OA period (1990-2000) and the post-OA period (2001-2004). CONCLUSIONS Open access was associated with increase in the number of citations received by the articles. It also decreased the lag time between publication and the first citation. For smaller biomedical journals, OA could be one of the means for improving visibility and thus citation rates."

Added 27 September 2005 Zhao, D. (2005)
Challenges of scholarly publications on the Web to the evaluation of science -- A comparison of author visibility on the Web and in print journals (abstract only)
Information Processing and Management, 41:6, 1403-1418, December 2005
Compares author visibility between the Web and print journals as revealed from citation analysis based on a search for the term "XML" or "eXtensible Markup Language" using NEC Research Institute’s CiteSeer, the entire ISI Science Citation Index (SCI) database, and journals indexed and classified in SCI as representing computer science research. The main finding: "The author ranking by number of citations that resulted from CiteSeer data is highly correlated with that obtained from SCI." i.e. it's not comparing OA impact vs non-OA but Web vs journal, and finds that authors, notably the top authors, are self-archiving and publishing papers in both places.

Added 11 February 2008 Coats, A. J. S. (2005)
Top of the charts: download versus citations in the International Journal of Cardiology (full-text requires subscription; otherwise abstract only)
International Journal of Cardiology, Volume 105, Issue 2, 2 November 2005, 123-125, available online 7 October 2005
From the abstract: "We have recorded the 10 top cited articles over a 12-month period and compared them to the 10 most popular articles being downloaded over the same time period. The citation-based listing included basic and applied, observational and interventional original research reports. For downloaded articles, which have shown a dramatic increase for the International Journal of Cardiology from 48,000 in 2002 to 120,000 in 2003 to 200,000 in 2004, the most popular articles over the same period are very different and are dominated by up-to-date reviews of either cutting-edge topics (such as the potential of stem cells) or of the management of rare or unusual conditions. There is no overlap between the two lists despite covering exactly the same 12-month period and using measures of peer esteem. Perhaps the time has come to look at the usage of articles rather than, or in addition to, their referencing."

Added 13 July 2005 Adams, J. (2005)
Early citation counts correlate with accumulated impact (abstract only)
Scientometrics, 63 (3): 567-581, June 2005
Working towards earlier prediction of impact. This paper is not OA and has just appeared but was written before Brody and Harnad (2005) revealed a correlation to predict impact from even earlier data, i.e. download data for OA papers, before any citations.

Added 26 September 2005 Moed, H. F. (2005)
Statistical Relationships Between Downloads and Citations at the Level of Individual Documents Within a Single Journal (abstract only)
Journal of the American Society for Information Science and Technology, 56(10): 1088-1097, published online 31 May 2005
"Statistical relationships between downloads from ScienceDirect of documents in Elsevier's electronic journal Tetrahedron Letters and citations to these documents recorded in journals processed by the (ISI) for the Science Citation Index (SCI) are examined. ... Findings suggest that initial downloads and citations relate to distinct phases in the process of collecting and processing relevant scientific information that eventually leads to the publication of a journal article." Does not investigate open access sources. Notes the need for caution in drawing conclusions on the frequency of paper downloads from formal citation patterns, and vice versa.

Updated 05 October 2005 Vaughan, L. and Shaw, D. (2005)
Web citation data for impact assessment: A comparison of four science disciplines (abstract only)
Journal of the American Society for Information Science and Technology, Vol. 56, No. 10, 1075 - 1087, published online 27 May 2005
appears to be an expansion of Can Web Citations be a Measure of Impact? An Investigation of Journals in the Life Sciences (abstract only)
ASIST 2004: Proceedings of the 67th ASIS&T Annual Meeting, Vol. 41 (Medford, USA: Information Today), pp. 516-526

Brody, T., Harnad, S. and Carr, L. (2005)
Earlier Web Usage Statistics as Predictors of Later Citation Impact
Author eprint, 18 May 2005, University of Southampton, School of Electronics and Computer Science, Journal of the American Association for Information Science and Technology, Volume 57, Issue 8, 2006, 1060-1072 (abstract)

Added 19 May 2005 Wren, J. D. (2005)
Open access and openly accessible: a study of scientific publications shared via the internet
BMJ, 330:1128, 12 April 2005

BMJ Rapid Responses to this article:
Carr, L., 15 April 2005 Open Access Misdefined "by misdefining Open Access this paper somehow ignores both the phenomenon that is being measured (Open Access-ibility) and the significant research that has risen around it."
Chan, C-h. and Ng, D. K., 13 May 2005 Technological problems in this study: PDF "The selection of only PDF reprint in this study made this correlation (between the time since initial publication and the probability of availability of reprint in non journal website) a little bit artificial."
All Rapid Responses

Wren's article also prompted this editorial
Suber, P. (2005)
Open access, impact, and demand
BMJ, 330:1097-1098, 14 May 2005

Added 13 April 2005 Belew, R. (2005)
Scientific impact quantity and quality: Analysis of two sources of bibliographic data (pdf 12pp)
Arxiv.org, cs.IR/0504036, 11 April 2005

Added 28 July 2008 De Groote, S. L., Shultz, M. and Doranski, M. (2005)
Online journals' impact on the citation patterns of medical faculty J Med Libr Assoc., 93 (2): 223–228, April 2005
From the conclusion: "It is possible that electronic access to information (i.e., online databases) has had a positive impact on the number of articles faculty will cite. Results of this study suggest, at this point, that faculty are still accessing the print-only collection, at least for research purposes, and are therefore not sacrificing quality for convenience."

Added 03 August 2006 Metcalfe, T. S. (2005)
The Rise and Citation Impact of astro-ph in Major Journals
ArXiv, Astrophysics, astro-ph/0503519, 23 March 2005
"I describe a simple method to determine the adoption rate and citation impact of astro-ph over time for any journal using NASA's Astrophysics Data System (ADS). I use the ADS to document the rise in the adoption of astro-ph for three major astronomy journals, and to conduct a broad survey of the citation impact of astro-ph in 13 different journals. I find that the factor of two boost in citations for astro-ph papers is a common feature across most of the major astronomy journals."

Updated 13 April 2005 Ongoing studies Hajjem, C. (2004-05)
Cover page for the range of studies highlighted below, Laboratoire de recherche en Sciences Cognitives, UQAM. (Text in French but graphs "self-explanatory"; see this comment for elaboration)

Évaluation de l'exactitude des résultats donnés par le robot (10 March 2005)
Crawler algorithms: the OA advantage is not just an artifact of the algorithm used to trawl the web to look for the full-text version of articles
Analyse de la variation de pourcentages d'articles en accès libre en fonction de taux de citations (8 March 2005)
Quality/impact self-selection bias: the OA advantage is *not* due only or largely to a quality/impact self-selection bias
Analyse de la variation de l’impact de citations des articles en accès libre par rapport aux articles disponibles uniquement par abonnement dans les disciplines biologie et sociologie par pays (undated)
National and discipline differences: there appear to be no systematic national differences, nor are there systematic differences between disciplines in the practise of OA provision
Variation de l’avantage de l’impact de citations des articles en accès libre (dans la discipline biologie) (29 November 2004)
On the OA advantage in biology
Variation de l’avantage de l’impact de citations des articles en accès libre dans la discipline sociologie (20 October 2004)
On the OA advantage in the social sciences

Updated 26 September 2005 Bollen, J., Van de Sompel, H., Smith, J. and Luce, R. (2005)
Toward alternative metrics of journal impact: A comparison of download and citation data (pdf 34pp)
Arxiv.org, cs.DL/0503007, 03 March 2005, in Information Processing and Management, 41(6): 1419-1440, December 2005

Added 5 January 2005 Ongoing study Brody, T., et al.
Citation Impact of Open Access Articles vs. Articles Available Only Through Subscription ("Toll-Access")
with downloadable graphs of '% Articles OA' and '% OA Advantage' by discipline and sub-discipline

Updated 31 January 2005 Schwarz, G. and Kennicutt Jr., R. C. (2004)
Demographic and Citation Trends in Astrophysical Journal Papers and Preprints (pdf 14pp)
Arxiv.org, astro-ph/0411275, 10 November 2004, Bulletin of the American Astronomical Society, Vol. 36, 1654-1663
See also a note from AAS Pub Board meeting, Tucson, November 3-4 2003
"Greg Schwarz (from the ApJ editorial office) reported some work he's doing tracking citation rates of papers published in the ApJ based on whether they were posted on astro-ph or not: ApJ papers that were also on astro-ph have a citation rate that is _twice_ that of papers not on the preprint server"
http://listserv.nd.edu/cgi-bin/wa?A2=ind0311&L=pamnet&D=1&O=D&P=1632

Added 03 August 2006 Havemann, F. (2004)
Eprints in der wissenschaftlichen kommunikation (Eprints in scientific communication)
Author eprint, 26 October 2004, presented at the Institute of Library Science, Humboldt University, Berlin, June 1, 2004
"the use of eprints can significantly accelerate the scientific communication. This was demonstrated by me with a small sample of articles in theoretical High Energy Physics published 1998 and 1999 in Physical Review D. Typically the eprints in this sample are available eight months before the printed issue is published. Three quarters of them are cited in eprints authored by other researchers before the journal issue appears (among them all highly cited eprints)."

Brody, T. (2004)
Citation Analysis in the Open Access World
Author eprint, October 4, 2004, in Interactive Media International

Added 9 November 2004 McVeigh, M. E. (2004)
Open Access Journals in the ISI Citation Databases: Analysis of Impact Factors and Citation Patterns
Thomson Scientific, October 2004

Added 29 September 2004 Antelman, K. (2004)
Do Open-Access Articles Have a Greater Research Impact?
College and Research Libraries, 65(5):372-382, September 2004
also Author eprint, E-LIS, 29 September 2004, http://eprints.rclis.org/archive/00002309/

Review of this article:
Lewis, S. P. (2006) Open Access Articles Have a Greater Research Impact Than Articles Not Freely Available, Evidence Based Library and Information Practice, 2006, 1:3

Updated 5 January 2005 Harnad, S., Brody, T., Vallieres, F., Carr, L., Hitchcock, S., Gingras, Y., Oppenheim, C., Stamerjohanns, H. and Hilf, E. (2004)
The Access/Impact Problem and the Green and Gold Roads to Open Access
Author eprint, 15 September 2004, in Serials Review, Vol. 30, No. 4, 310-314 (free access to published version during 2005)
Shorter version: The green and the gold roads to Open Access
Nature, Web Focus: access to the literature, May 17, 2004

Updated 26 September 2005 Kurtz, M. J., Eichhorn, G., Accomazzi, A., Grant, C. S., Demleitner, M., Murray, S. S. (2004b)
The Effect of Use and Access on Citations
Author eprint, September 2004, in Information Processing and Management, 41 (6): 1395-1402, December 2005

Comment and discussion:
Moed, H. (2006) reply on OA to sigmetrics, SIGMETRICS list server, 8 December 2006
Harnad, S. (2006) The Self-Archiving Impact Advantage: Quality Advantage or Quality Bias?, Open Access Archivangelism, 20 November 2006
Harnad, S. (2006) The Special Case of Astronomy, Open Access Archivangelism, October 14. 2006

Perneger, T. V. (2004)
Relation between online "hit counts" and subsequent citations: prospective study of research papers in the BMJ
BMJ, 329:546-547, 4 September 2004

BMJ Rapid Responses to this article:
Harnad, S. and Brody, T., 6 September 2004 Prior evidence that downloads predict citations "confirms what Tim Brody's online usage/citation correlator has been demonstrating for several years now across a number of areas in physics and mathematics.
All Rapid Responses

Prakasan, E. R. and Kalyane, V. L. (2004)
Citation analysis of LANL High-Energy Physics E-Prints through Science Citation Index (1991-2002)
Author eprint, E-LIS, 26 August 2004

Added 13 April 2005 Murali, N. S., Murali, H. R., Auethavekiat, P., Erwin, P. J., Mandrekar, J. N., Manek, N. J. and Ghosh, A. K. (2004)
Impact of FUTON and NAA Bias on Visibility of Research
Mayo Clinic Proceedings, Vol. 79, No. 8, 1001-1006, August 2004
Notes and comment: FUTON = full text on the Net; NAA = no abstract available
This is not an article on how Open Access increases impact but on how *Online* Access increases impact. The effects are related, but one is a licensing effect, not an OA effect.

Added 10 May 2007 Davis, P. M. (2004)
For Electronic Journals, Total Downloads Can Predict Number of Users
portal: Libraries and the Academy, Vol. 4, No. 3, July 2004, 379–392

Harnad, S. and Brody, T. (2004a)
Comparing the Impact of Open Access (OA) vs. Non-OA Articles in the Same Journals
D-Lib Magazine, Vol. 10 No. 6, June 2004
Replicates the Lawrence effect -- OA increases impact -- in physics.

Pringle, J. (2004)
Do Open Access Journals have Impact?
Nature, Web Focus: access to the literature, May 7, 2004

Testa, J. and McVeigh, M. E. (2004)
The Impact of Open Access Journals: A Citation Study from Thomson ISI (pdf 17pp)
Author eprint, 14 April 2004

Kurtz, M. J. (2004)
Restrictive access policies cut readership of electronic research journal articles by a factor of two (pdf 2pp)
Harvard-Smithsonian Centre for Astrophysics, Cambridge, MA
Poster presentation at National Policies on Open Access (OA) Provision for University Research Output: an International meeting, Southampton, 19 February 2004

Brody, T., Stamerjohanns, H., Harnad, S., Gingras, Y. and Oppenheim, C. (2004)
The Effect of Open Access on Citation Impact (pdf 1pp)
Poster presentation at National Policies on Open Access (OA) Provision for University Research Output: an International meeting, Southampton, 19 February 2004

Updated 5 January 2005 Kurtz, M. J., Eichhorn, G., Accomazzi, A., Grant, C. S., Demleitner, M. and Murray, S. S. (2004a)
Worldwide Use and Impact of the Nasa Astrophysics Data System Digital Library
Author eprint, January 28, 2004, in Journal of the American Society for Information Science and Technology, Vol. 56, No. 1, 36-45, published online 20 September 2004

Hitchcock, S., Brody, T., Gutteridge, C., Carr, L. and Harnad, S. (2003b)
The Impact of OAI-based Search on Access to Research Journal Papers
Author eprint, 15 September 2003, in Serials, Vol. 16, No. 3, November 2003, 255-260

Hitchcock, S., Woukeu, A., Brody, T., Carr, L., Hall, W. and Harnad, S. (2003a)
Evaluating Citebase, an open access Web-based citation-ranked search and impact discovery service
Technical Report ECSTR-IAM03-005, School of Electronics and Computer Science, University of Southampton, July 2003

Added 28 October 2004 Bollen, J., Vemulapalli, S. S., Xu, W. and Luce, R. (2003)
Usage Analysis for the Identification of Research Trends in Digital Libraries
D-Lib Magazine, Vol. 9, No. 5, May 2003

Kurtz, M. J., Eichhorn, G., Accomazzi, A., Grant, C., Demleitner, M., Murray, S. S., Martimbeau, N. and Elwell, B. (2003b)
The NASA Astrophysics Data System: Sociology, Bibliometrics, and Impact
Author eprint, March 2003, Journal of the American Society for Information Science and Technology, submitted for publication

Updated 23 February 2005 Kurtz, M. J., Eichhorn, G., Accomazzi, A., Grant, C. S., Demleitner, M., Murray, S. S., Martimbeau, N. and Elwell, B. (2003a)
The Bibliometric Properties of Article Readership Information
Author eprint, March 2003, in Journal of the American Society for Information Science and Technology, 56 (2): 111-128, January 15, 2005

Added 17 December 2007 Drenth, J. P. H. (2003)
More reprint requests, more citations? (subscriber access to full text)
Scientometrics, Vol. 56, No. 2, February 2003, 283-286, revised version published online August 2006
From the abstract: "This study aims to correlate the number of reprint requests from a 10-year-sample of articles with the number of citations. ... Articles that received most reprint requests are cited more often."

Darmoni, S. J., et al. (2002)
Reading factor: a new bibliometric criterion for managing digital libraries
Journal of the Medical Library Association, Vol. 90, No. 3, July 2002

Kurtz, M. J., Eichhorn, G., Accomazzi, A., Grant, C. S., Thompson, D. M., Bohlen, E. H. and Murray, S. S. (2002)
The NASA Astrophysics Data System: Obsolescence of Reads and Cites (pdf 8pp)
Library and Information Services in Astronomy IV, edited by B. Corbin, E. Bryson, and M. Wolf, July 2002

Bollen, J. and Luce, R. (2002)
Evaluation of Digital Library Impact and User Communities by Analysis of Usage Patterns
D-Lib Magazine, Vol. 8, No. 6, June 2002

Lawrence, S. (2001)
Free online availability substantially increases a paper's impact
Nature, 31 May 2001

Added 04 October 2005 Anderson, K., Sack, J., Krauss, L. and O'Keefe, L. (2001)
Publishing Online-Only Peer-Reviewed Biomedical Literature: Three Years of Citation, Author Perception, and Usage Experience
Journal of Electronic Publishing, Vol. 6, No. 3, March 2001
One of the first studies of the citation effect of online against offline publication, rather than of open access against non-OA. Provides data for one journal and a small number of articles over a three year period year. This paper was added to the bibliography following this correspondence:

Banks, P. "does not find the same citation advantage for online publications claimed by Harnad and his colleagues." Harnad, S. "Please weigh findings against the preponderance of evidence."
American Scientist Open Access Forum, Open access to research worth £ 1.5bn a year, 28 September 2005

Updated 04 October 2005 Odlyzko, A. M. (2000)
The rapid evolution of scholarly communication
PEAK 2000: Economics and Usage of Digital Library Collections conference, Ann Arbor, MI, March 2000.
Also in Learned Publishing, 15(1), 7-19, January 2002 here. Author eprint http://www.dtc.umn.edu/~odlyzko/doc/rapid.evolution.pdf
Notes the growing usage of information in electronic form (c.f. print forms) and of journal papers from non-journal sites (e.g. eprints), and presents evidence that usage increases when access is more convenient

Youngen, G. K. (1998)
Citation Patterns to Electronic Preprints in the Astronomy and Astrophysics Literature
Library and Information Services in Astronomy III, ASP Conference Series, Vol. 153, 1998
see also
Citation Patterns to Traditional and Electronic Preprints in the Published Literature
College & Research Libraries, September 1998

Youngen, G. (1998)
Citation Patterns Of The Physics Preprint Literature With Special Emphasis On The Preprints Available Electronically
Author eprint, UIUC Physics and Astronomy library, c. 5 November 1998, presented at ACRL/STS on 6/29/97

Web tools for measuring impact

Citebase Search "Search and citation analysis tool for the free, online research literature" http://citebase.eprints.org/ User service, free
see
Added 28 July 2008 Jacsó, P. (2004) CiteBase Search, Online, Sep/Oct 2004, 57-58
Brody, T. (2003) Citebase Search: Autonomous Citation Database for e-print Archives, sinn03 conference on Worldwide Coherent Workforce, Satisfied Users - New Services For Scientific Information, Oldenburg, Germany, September 2003
Hitchcock, S., et al. (2003a) Evaluating Citebase, an open access Web-based citation-ranked search and impact discovery service

Correlation Generator http://citebase.eprints.org/analysis/correlation.php
Generates a graph (or table) of the correlation between citation impact and usage impact from the Citebase database
see Brody, T. and Harnad, S. 2005 (in prep.)

Citeseer "Scientific literature digital library" http://citeseer.ist.psu.edu/ User service, free
Updated 26 May 2008 Now available as CiteSeer^x, or "Next Generation CiteSeer" http://citeseerx.ist.psu.edu
see
Jacsó, P., (2005) CiteSeer, Thomson Gale, November 2005
Lawrence, S., Giles, C. L., Bollacker, K. (1999), Digital Libraries and Autonomous Citation Indexing, IEEE Computer, Vol. 32, No. 6, 67-71, 1999

Elsevier Scopus Bibliographic database covering 13,450 peer-reviewed titles http://www.scopus.com/ User service
see
Added 28 July 2008 Jacsó, P. (2007) Scopus (2008 Winter Release), Gale, Reference Reviews, Péter's Digital Reference Shelf, November 2007
Added 15 March 2006 Burnham, J. F. (2006) Scopus database: a review, Biomedical Digital Libraries, 3:1, 8 March 2006
Added 15 March 2006 Dess, H. M. (2006) Scopus, Issues in Science and Technology Librarianship, Winter 2006
Added 15 May 2006 Quint, B. (2006) Elsevier’s Scopus Introduces Citation Tracker: Challenge to Thomson ISI’s Web of Science?, Newsbreaks, January 23, 2006
Added 13 September 2007 Goodman, D. and Deis, L. (2006) Update on Scopus, The Charleston Advisor, Vol. 7, No. 3, January 2006, 42-43
Jacsó, P. (2004) Scopus, Thomson Gale, September 2004
see also Comparative reviews

Google Scholar Find articles from academic publishers, preprint repositories and universities, as well as scholarly articles across the web (presents citations as separate results) http://scholar.google.com/ User service, free
see

Added 28 July 2008 Harzing, A. W.K., van der Wal, R. (2008) Google Scholar as a new source for citation analysis, Ethics in Science and Environmental Politics, Vol. 8, No. 1, June 03, 2008, 61-73
Added 28 July 2008 Jacsó, P. (2008) The pros and cons of computing the h-index using Google Scholar. Online Information Review, 32(3) 2008, 437-452
Added 26 May 2008 Meier, J. J. and Conkling, T. W. (2008) Google Scholar's Coverage of the Engineering Literature: An Empirical Study, Journal of Academic Librarianship, Vol. 34, No. 3, May 2008, 196-201 (full text requires subscription; abstract only)
Added 28 July 2008 Jacsó, P. (2008) Google Scholar, Online, Mar/Apr 2008, 53-54
Added 13 September 2007 Harzing, A.-W. (2007) Reflections on Google Scholar, Harzing.com, fifth version, 6 September 2007
about the citation analysis software Publish or Perish and its relation with Google Scholar
Added 13 September 2007 Quint, B. (2007) Changes at Google Scholar: A Conversation With Anurag Acharya, NewsBreaks, August 27, 2007
rare public interview with the low-profile 'designer and missionary' behind Google Scholar
Added 22 August 2007 Mayr, P. and Walter, A.-K. (2007) An exploratory study of Google Scholar, arXiv.org > cs > arXiv:0707.3575v1 [cs.DL], July 24, 2007, in Online Information Review, Vol. 31, No. 6 (2007), to appear. Author preprint also available from http://www.ib.hu-berlin.de/~mayr/arbeiten/OIR-Mayr-Walter-2007.pdf
Added 10 May 2007 Robinson, M. L. and Wusteman, J. (2007) Putting Google Scholar to the test: a preliminary study, author eprint, also in Program: Electronic Library and Information Systems, Vol. 41, Issue 1, February 2007, 71-80
Added 15 May 2006 Sadeh, T. (2006) Google Scholar Versus Metasearch Systems, HEP Libraries Webzine, issue 12, March 2006
"thoughtful and informative ... altogether the best overview of Google Scholar, other large federated search systems such as Scirus, and library-based metasearch tools I've seen." Reviewed by Tennant, R., Current Cites, January 2006 issue
Added 15 March 2006 Burright, M. (2006) Google Scholar -- Science & Technology, Issues in Science and Technology Librarianship, Winter 2006
Added 28 February 2006 Noruzi, A. (2005) Google Scholar: the new generation of citation indexes (pdf 11pp), E-LIS, 11 February 2006, in LIBRI 55(4): 170-180
Jacsó, P., (2005) Google Scholar and The Scientist, commenting on his interview in Perkel, J., The Future of Citation Analysis (abstract only), The Scientist, October 24, 2005
Jacsó, P. (2005) Google Scholar (Redux), Thomson Gale, June 2005
Myhill, M. (2005) Google Scholar, Charleston Advisor, Vol. 6, No. 4, April 2005
Added 22 August 2007 Giustini, D. and Barsky, E. (2005) A look at Google Scholar, PubMed, and Scirus: comparisons and recommendations, Journal of the Canadian Health Libraries Association/Journal de l'Association des bibliothèques de la santé du Canada (JCHLA / JABSC) 26: 85-89 (2005) (pdf 5pp)
Jacsó, P. (2004) Google Scholar Beta, Thomson Gale, December 2004
see also Comparative reviews

ISI Web of Science Cited reference searching of 8,700 high impact research journals http://www.isinet.com/products/citation/wos/ User service
see
Added 28 July 2008 Jacsó, P. (2007) Web of Science, Gale, Reference Reviews, Péter's Digital Reference Shelf, January 2007
Jacsó, P. (2004) Web of Science Citation Indexes, Thomson Gale, August 2004
see also Comparative reviews

Added 11 December 2006 Rexa.Info Covers the computer science research literature. Rexa is "a sibling to CiteSeer, Google Scholar, Academic.live.com, the ACM Portal. It's chief enhancement is that Rexa knows about more first-class, de-duplicated, cross-referenced object types: not only papers and their citation links, but also people, grants, topics" http://rexa.info/ User service, free (login required)

Added 13 April 2006 Windows Live Search Academic Beta version. Indexes content related to computer science, physics, electrical engineering, and related subject areas, with more than 6 million records from approximately 4300 journals, 2000 conferences and ArXiv.org. In collaboration with Citeseer http://academic.live.com/ User service, free
see
Added 26 May 2008 Nadella, S. (2008) Book search winding down, Live Search, The official blog of the Live Search team at Microsoft, May 23, 2008
"Live Search Books and Live Search Academic projects ... will be taken down next week. Books and scholarly publications will continue to be integrated into our Search results, but not through separate indexes."
Added 28 July 2008 Jacsó, P. (2008) Live Search Academic, Gale, Reference Reviews, Péter's Digital Reference Shelf, April 2008
Added 28 July 2008 Jacsó, P. (2006) Windows Live Academic, Online, Sep/Oct 2006, 59-60
Added 15 May 2006 Quint, B. (2006) Windows Live Academic Search: The Details, Newsbreaks, April 17, 2006
Added 13 April 2006 Sherman, C. (2006) Microsoft Launches Windows Live Academic Search, SearchEngineWatch.com, April 12, 2006

Added 15 May 2006 Citations in Economics not intended for direct user access; instead is made available to RePEc services such as Socionet, EconPapers and IDEAS. Uses Citeseer software http://citec.repec.org/ Data service, free
Rank working papers series and journals in Economics http://citec.repec.org/s/
see
Barrueco Cruz, J. M. and Krichel, T. (2004) Building an autonomous citation index for grey literature: the economics working papers case (pdf 12pp), E-LIS, 01 February 2005, also in Proceedings GL6: Sixth International Conference on Grey Literature, New York, December 2004

CrossRef Forward linking service tool allows CrossRef member publishers to display cited-by links in their primary content, Data service
CrossRef and Atypon announce forward linking service (press release) June 8, 2004
Institute of Physics becomes first journals publisher to implement 'cited-by' links using CrossRef's Forward Linking service: Time travel with IOP journals (IOP press release) 14 March, 2005

Forthcoming ISI Web Citation Index User service
see
Added 15 May 2006 Martello, A. (2006) Selection of Content for the Web Citation Index: Institutional Repositories and Subject-Specific Archives, Thomson.com, undated
Pringle, J. (2005) Partnering helps institutional repositories thrive, KnowledgeLink Newsletter, February 2005
Citeseer's replacement? List server mailing, 18 March 2004
Quint, B. (2004) Thomson ISI to Track Web-Based Scholarship with NEC’s CiteSeer, Information Today Newsbreaks, March 1, 2004

Comparative reviews

Added 28 July 2008 Jacsó, P. (2008)
The Plausibility of Computing the H-index of Scholarly Productivity and Impact Using Reference Enhanced Databases
Online Information Review, 32(2) 2008, 266-283
"aims to provide a general overview of the three largest, cited-reference-enhanced, multidisciplinary databases (Google Scholar, Scopus, and Web of Science) for determining the h-index. The practical aspects of determining the h-index also need scrutiny, because some content and software characteristics of reference-enhanced databases can strongly influence the h-index values."

Added 26 May 2008 Meho, L. I. and Rogers, Y. (2008) Citation Counting, Citation Ranking, and h-Index of Human-Computer Interaction Researchers: A Comparison between Scopus and Web of Science, E-LIS, 10 March 2008, in Journal of the American Society for Information Science and Technology, to appear

Added 26 May 2008 Kloda, L. A. (2007) Use Google Scholar, Scopus and Web of Science for Comprehensive Citation Tracking, Evidence Based Library and Information Practice, 2(3): 87-90, 2007, also in E-LIS, 21 September 2007 http://eprints.rclis.org/archive/00011437/

Added 26 May 2008 Schroeder, R. (2007) Pointing Users Toward Citation Searching: Using Google Scholar and Web of Science, portal: Libraries and the Academy, Vol. 7, No. 2, April 2007, 243-248 (full text requires subscription)

Added 13 September 2007 Goodman, D. and Deis, L. (2007) Update on Scopus and Web of Science, The Charleston Advisor, Vol. 8, No. 3, January 2007, 15-18

Added 10 May 2007 Meho, L. I. and Yang, K. (2006) A New Era in Citation and Bibliometric Analyses: Web of Science, Scopus, and Google Scholar, arXiv.org, Computer Science, cs/0612132, 23 Dec 2006, published as Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar, in Journal of the American Society for Information Science and Technology, Vol. 58, No. 13, 2007, 2105-2125

Added 11 December 2006 Fingerman, S. (2006) Web of Science and Scopus: Current Features and Capabilities, Issues in Science and Technology Librarianship, Fall, 2006

Added 17 January 2007 Neuhaus, C. and Daniel, H.-D. (2006) Data sources for performing citation analysis: An overview, ETH E-Collection, June 30, 2006, Journal of Documentation, accepted for publication
Reports the limitations of Thomson Scientific’s citation indexes and reviews the characteristics of the citation-enhanced databases Chemical Abstracts, Google Scholar and Scopus.

Comment on this paper:
Pikas, C. K. "I think this article will be very helpful, but the extension that seems necessary right now is to CrossRef data. Many publishers such as the Optical Society of America via Optics Infobase provide forward and backward citations using CrossRef. When trying to *approach* comprehensiveness, I felt I had to look there as well as Scopus, Web of Science, Google Scholar, and CA. In fact, I found many citations there that were unique -- but this is not scientific, merely anecdotal." Sigmetrics listserv, 14 Dec 2006

Bakkalbasi, N., Bauer, K., Glover, J. and Wang, L. (2006) Three options for citation tracking: Google Scholar, Scopus and Web of Science, Biomedical Digital Libraries, June 29, 2006

Added 8 March 2007 Bosman, J., van Mourik, I., Rasch, M., Sieverts, E. and Verhoeff, H. (2006) Scopus reviewed and compared, Igitur repository, Utrecht University, June 2006
The coverage and functionality of the citation database Scopus, including comparisons with Web of Science and Google Scholar

Wenzel, E. (2006) Google Scholar beta, ZDNet, May 2, 2006
Brief comparison of Google Scholar and Microsoft Live Academic Search

Bailey, C. W. Jr (2006) A Simple Search Hit Comparison for Google Scholar, OAIster, and Windows Live Academic Search, Digital Koans, author blog, April 13, 2006
A simple but revealing experiment: "It should be clear that a sample of one search term is a very crude measure".

Pauly, D. and Stergiou, K. I. (2005) Equivalence of results from two citation analyses: Thomson ISI’s Citation Index and Google’s Scholar service (pdf 3pp), Ethics in Science and Environmental Politics, 22 December 2005, 33-35

Comment on this paper:
""outperform" means many things. OK, in this study the citation counts were close but the searchability of material on WOS is much stronger." ResourceShelf, February 25, 2006

Added 23 November 2006 Jacso, P. (2005) Comparison and analysis of the citedness scores in Web of Science and Google Scholar (pdf 10pp), Digital Libraries: Implementing Strategies and Sharing Experiences, Lecture Notes In Computer Science, 3815: 360-369, 2005, Proceedings of the 8th International Conference on Asian Digital Libraries, ICADL 2005, Bangkok, Thailand, December 12-15, 2005

Roth, D. L. (2005) The emergence of competitors to the Science Citation Index and the Web of Science (pdf 6pp), Current Science Online, Vol. 89, No. 9, 10 November 2005

Jacsó, P. (2005) As we may search – Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases (pdf 11pp), Current Science Online, Vol. 89, No. 9, 10 November 2005

Bauer, K. and Bakkalbasi, N. (2005) An Examination of Citation Counts in a New Scholarly Communication Environment, D-Lib Magazine, 11(9), September 2005.
Compares citation counts provided by Web of Science, Scopus, and Google Scholar.

Comment on this paper:
Stegmann, J. Clearer picture? "the authors of this interesting paper should, perhaps, take into account the ISI Proceedings database because the conference papers indexed therein are included together with their references. ... This would give a clearer picture of what one gets from free services like Google Scholar and from products which have to be licensed." Sigmetrics listserv, 16 September 2005
Author response: Focused and cross-disciplinary "We decided to concentrate on three multi-disciplinary databases only, as a way of focusing our work. We wanted to lay the foundation for a study that will compare subject areas: hence our decision to look at databases that cross many subject areas." Sigmetrics listserv, 16 September 2005

LaGuardia, C. (2005) Scopus vs. Web of Science, Library Journal, 130(1); 40, 42, January 15, 2005

Deis, L. F. and Goodman, D. (2005) Web of Science (2004 version) and Scopus, Charleston Advisor, Vol. 6, No. 3, January 2005

Background

The financial imperative: correlating research access, impact and assessment

There is another dimension to the open access advantage. If open access increases impact, then it will also increase research income and funding. It has been shown in the UK that there is a correlation between research assessment ratings and citation counts, and higher ratings means more money for the higher rated research groups. Of course, if all papers were made open access by their authors, the relative effect would disappear. First-mover advantage anyone?

"Research impact translates into money: employment, salary, tenure money, as well as research-funding money: (1) RAE rank correlates with substantial top-sliced funding, (2) it also correlates highly (0.91) with citation counts, and (3) self-archiving increases citation counts by 50-250+%. Do you really think that any researcher who is *aware* of those three correlations is being rational if he doesn't self-archive?" Stevan Harnad

Added 10 November 2008 Oppenheim, C. and Summers, M. A. C.
Citation counts and the Research Assessment Exercise, part VI: Unit of assessment 67 (music)
Information Research, 13 (2), paper 342, June 2008

Added 28 July 2008 Adler, R., Ewing, J. (Chair) and Taylor, P. (2008)
Citation Statistics (pdf 26pp)
Joint Committee on Quantitative Assessment of Research, International Mathematical Union, IMU-ICIAM-IMS, 6/11/2008

Comment on this paper:

Citation statistics, American Scientist Open Access Forum
Armbruster, C., 11 June 2008: "after reading the report, I would caution against dismissing it. Science and scientists should be concerned about the politicisation of metrics. Politicisation comes from governments and research funders but is also going on inside academic institutions. Moreover, in a general sense the citation and usage metrics currently available are not 'fit for purpose'. Worse still, politicisation carries with it the significant risk of arresting the development of tools for metric research evaluation. ... All we have at the moment are some 'quick fix metrics'. And these are increasingly used to make and legitimate all kinds of decisions. It is thus welcome that mathematicians and statisticians scrutinise current practices and show up the lack of validity and reliability of many measures, technical faults as well as the misguided judgements of peers, university management, funding agencies and government."
Oppenheim, C., 12 June 2008: "it fails to address the fundamental issue, which is: citation and other metrics correlate superbly with subjective peer review. Both methods have their faults, but they are clearly measuring the same (or closely related) things. Ergo, if you have evaluate research in some way, there is no reason NOT to use them! It also keeps referring to examples from the field of maths, which is a very strange subject citation-wise."

Harnad, S., Citation Statistics: International Mathematical Union Report, Open Access Archivangelism blog, June 15, 2008: "what all this valuable, valid cautionary discussion overlooks is not only the possibility but the empirically demonstrated fact that there exist metrics that are highly correlated with human expert rankings. It follows that to the degree that such metrics account for the same variance, they can substitute for the human rankings. The substitution is desirable, because expert rankings are extremely costly in terms of expert time and resources. Moreover, a metric that can be shown to be highly correlated with an already validated variable predictor variable (such as expert rankings) thereby itself becomes a validated predictor variable. And this is why the answer to the basic question of whether the RAE's decision to convert to metrics was a sound one is: Yes."
Bensman, S. J., IMU Critique of Citation Analysis, SIGMETRICS, 27 June 2008: "I checked the distribution of mathematics journals by impact factor in the 2007 SCI JCR. It was as I suspected. The range of impact factors was only from 0.108 to 2.739--extraordinarily low and tight--and the top journals on the impact factor had no review articles. This is suggestive of an extremely random citation pattern with no development of consensual paradigms. Therefore, math acts like a humanities in terms of its literature use, and citation analysis is probably not applicable to this discipline. If citation analysis is used, it has to be backed by other measures."

Added 28 July 2008 Harnad, S. (2008)
Validating research performance metrics against peer rankings
Ethics in Science and Environmental Politics, Vol. 8, No. 1, June 03, 2008, 103-107

Added 11 February 2008 Armbruster, C. (2008)
Access, Usage and Citation Metrics: What Function for Digital Libraries and Repositories in Research Evaluation?
Social Science Research Network, February 05, 2008
From the abstract: "This systematic appraisal of the future role of digital libraries and repositories for metric research evaluation proceeds by investigating the practical inadequacies of current metric evaluation before defining the scope for libraries and repositories as new players. Service reviewed include: Leiden Ranking, Webometrics Ranking of World Universities, COUNTER, MESUR, Harzing POP, CiteSeer, Citebase, RePEc LogEc and CitEc, Scopus, Web of Science and Google Scholar."

Added 26 May 2008 Surya, M., D'Este, P. and Neely, A. (2008)
Citation Counts: Are They Good Predictors of RAE Scores? A bibliometric analysis of RAE 2001
Cranfield QUEprints, 31.01.2008

Comment on this paper:
RAE/REF research, JISC-REPOSITORIES, 15 February 2008
Oppenheim, C. "I've had a first look at it; it uses methods not previously employed in such studies and without a full explanation of how the research was conducted. That's not to say it is invalid, but it is a lost opportunity, having collected so much data, not to have followed standard methods or to explain things more fully."
Johnson, I. M. "shows an inadequate understanding of the contextual issues. Moreover, most of the discussion in section 5 appears to have little or no basis in the preceding data."
Harnad, S. "This pilot study has some methodological weaknesses. The remedy for the ostensible problems encountered in this study is for the panel rankings in the parallel metric/panel RAE 2008 to be analysed in a full-scale multiple regression study using as rich and diverse as possible a spectrum of predictive metrics (not just citation counts!), discipline by discipline."

Added 10 May 2007 Harnad, S. (2007)
Open Access Scientometrics and the UK Research Assessment Exercise
ArXiv, Computer Science, cs.IR/0703131, 26 March 2007. Preprint of invited keynote address to 11th Annual Meeting of the International Society for Scientometrics and Informetrics, Madrid, 25-27 June 2007
also in ECS EPrints, 29 March 2007 http://eprints.ecs.soton.ac.uk/13804/

Added 19 November 2006 Steele, C., Butler, L. and Kingsley, D. (2006)
The publishing imperative: the pervasive influence of publication metrics
ANU Institutional Repository, 30 October 2006, also in Learned Publishing, 19(4): 277-290, October 2006

Added 19 November 2006 Houghton J. and Sheehan, P. (2006)
The Economic Impact of Enhanced Access to Research Findings
Centre for Strategic Economic Studies. Victoria University. July 2006
See also
Houghton, J., Steele, C. and Sheehan, P. (2006)
Research Communication Costs In Australia: Emerging Opportunities And Benefits
Department of Education, Science and Training (DEST), Australia, September 2006

Comment on The Economic Impact of Enhanced Access to Research Findings:
Harnad, S. Maximising the Return on Research "These estimates agree substantially with prior estimates that have been made (e.g., for the UK, Canada and Australia)." American-Scientist-Open-Access-Forum, 9 August 2006

Added 13 September 2007 Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006)
The Open Research Web: A Preview of the Optimal and the Inevitable
ECS EPrints, 02 May 2006, in Open Access: Key Strategic, Technical and Economic Aspects, Jacobs, N., Ed., chapter 21 (Oxford: Chandos Publishing)

Added 26 September 2005 Harnad, S. (2005)
Maximising the Return on UK's Public Investment in Research
Author eprint, September 14, 2005
Attempts to monetise 'lost' impact: "The online-age practice of self-archiving has been shown to increase citation impact by a dramatic 50-250%, but so far only 15% of researchers are doing it spontaneously. Citation impact is rewarded by universities (through promotions and salary increases) and by research-funders like RCUK (through grant funding and renewal) at a conservative estimate of £46 per citation. ... As a proportion of the RCUK’s yearly £3.5bn research expenditure (yielding 130,000 articles x 5.6 = 761,600 citations), our conservative estimate would be 50% x 85% x £3.5.bn = £1.5bn worth of loss in potential research impact (323,680 potential citations lost)."
See also Australia is not maximisng the return on its research investment (ETD2005, Sydney) for the same estimate applied to the potential lost return ($425M) there.

Comment on this paper:
Rowland, I. "If I am the one millionth author (or the 10,000th research group or the 100th nation) to publish open access, that comparative advantage must quickly decline, approaching zero as the last few laggards pile in". Author response: "Ian Rowland is exactly right that the OA impact advantage (currently 50-250%) will shrink as we approach 100% OA. Right now we are at 15% OA, and the advantage is in part -- no one can say how large a part -- a *competitive* advantage of the minority 15% OA -- the head-start vanguard -- over the laggard 85% non-OA majority. ... The OA impact advantage consists of at least the following 6 factors ..."
American Scientist Open Access Forum, OA advantage = EA + (AA) + (QB) + QA + (CA) + UA, 15 September 2005
Waters, D. "Mr. Harnad may well be on to an important subject and line of argument in suggesting that citations are a kind of return on investment. However, close inspection of the concepts and logic of his argument suggests that he is quite a bit further from proving his case than he seems to have convinced himself that he is." Author response: "No proof here: Just conservative estimates."
American Scientist Open Access Forum, Open access to research worth 1.5bn a year, 27 September 2005

Day, M. (2004)
Institutional repositories and research assessment (pdf 29pp)
Author eprint (v. 0.1), 2 December 2004

Harnad, S. (2003)
Maximizing university research impact through self-archiving
Jekyll.com, No. 7, December 2003

Harnad, S. (2003)
Enhance UK research impact and assessment by making the RAE webmetric
Author eprint, in Times Higher Education Supplement, 6 June 2003, p. 16

Harnad, S., Carr, L., Brody, T. and Oppenheim, C. (2003)
Mandated online RAE CVs linked to university eprint archives: Enhancing UK research impact and assessment
Ariadne, issue 35, April 2003

Smith, A. and Eysenck, M. (2002)
The correlation between RAE ratings and citation counts in psychology (pdf 12pp)
Technical Report, Psychology, Royal Holloway College, University of London, June 2002

Holmes, A. and Oppenheim, C. (2001)
Use of citation analysis to predict the outcome of the 2001 Research Assessment Exercise for Unit of Assessment (UoA) 61: Library and Information Management
Information Research, Vol. 6, No. 2, January 2001

Harnad, S. (2001)
Research Access, Impact and Assessment (longer version)
Author eprint, in Times Higher Education Supplement, 1487: p. 16., 2001

Garfield, E. (1988)
Can Researchers Bank on Citation Analysis? (pdf 10pp)
Current Comments, No. 44, October 31, 1988
attached (pp 3-10)
Diamond, Jr., A. M. (1986)
What is a Citation Worth?
J. Hum. Resour., 21:200-15, 1986
Garfield comments on studies that attempt to quantify the reward system of science in terms of monetary returns to author salaries from article publication and citations, reprinting one of those studies

Citation analysis, indexes and impact factors

Notes. Important work in this area builds on Eugene Garfield's pioneering work in the 1950s. Although producing some remarkably successful tools for measuring the impact of the scholarly literature, this area is not without controversy. This short list presents a cross-section underlining these issues, with a view to understanding how such long-established approaches might adapt to online data, and how possible shortcomings might be overcome.

Added 10 November 2008 Leydesdorff, L. (2008)
How are new citation-based journal indicators adding to the bibliometric toolbox?
Author preprint, undated, (announced 31 Oct 2008)

Added 10 November 2008 Radicchi, F., Fortunato, S. and Castellano, C. (2008)
Universality of citation distributions: towards an objective measure of scientific impact
arXiv.org, arXiv:0806.0974v2 [physics.soc-ph], 5 Jun 2008 (v1), last revised 27 Oct 2008

Comment on this paper:
Davis, P., Universal Citations, the scholarly kitchen blog, Nov 3, 2008: "differences between disciplines can be quickly remediated by a simple, intuitive calculation: divide the number of citations to a paper by the average number of citations to all papers in its discipline for that year. The effect is stunning, and seems to hold irrespective of the publication year studied. Looking at the effect of this normalization is like looking at ducks lining up in a row."

Added 10 November 2008 Banks, M. A. and Dellavalle, R. (2008)
Emerging Alternatives to the Impact Factor
E-LIS, 05 September 2008, also in OCLC Systems & Services, 24(3)

Added 10 November 2008 Brumback, R. A. (2008)
Worshiping false idols: the impact factor dilemma
J. Child Neurol., Vol. 23, No. 4, April 2008, 365-367
"the opacity in Thomson Scientific's refusal to reveal the details of their calculations only serves to increase suspicion about possible data manipulations. ... Now would seem to be the appropriate time for the academic community to demand valid metrics to assess published scientific material"

Comment on this paper:
Pringle, J., Correcting the Record, J. Child Neurol., Vol. 23, No. 9, September 2008, 1092: "reiterates statements made by Michael Rossner about a supposed discrepancy in our database relating to the impact factor calculations for the Journal of Experimental Medicine. The Rossner editorial, though republished several times, each time repeats the same discredited assertions. I refer your readers to the corrections published by Thomson Scientific"
Brumback, R. A., Response to Correspondence, J. Child Neurol., Vol. 23, No. 9, September 2008, 1092-1094: "it is disappointing that Pringle chose to use typical faulty reasoning by attacking my citing of the article by Rossner et al (which was just 1 of the total 32 references) rather than addressing the real issues raised in my editorial. ... Concerns about the journal impact factor values are not new and have been voiced for more than a decade (but mostly to deaf ears)^8-38 ... Although it would be preferable to have journal indicators produced by an independent not-for-profit organization, the Thomson Scientific journal impact factor could be acceptable if Thomson Scientific were carefully to address each specific concern that has been raised and then provide a truly open transparent reporting system with resultant nonmanipulable values."

Added 26 May 2008 Althouse, B. M., West, J. D., Bergstrom, T. C. and Bergstrom, C. T. (2008)
Differences in Impact Factor Across Fields and Over Time
eScholarship Repository, California Digital library, Department of Economics, University of California Santa Barbara, Departmental Working Papers, paper 2008-4-23, April 23, 2008

Added 11 February 2008 Kosmopoulos, C. et Pumain, D. (2007)
Citation, Citation, Citation: Bibliometrics, the web and the Social Sciences and Humanities
Cybergeo, Science et Toile, article 411, mis en ligne le 17 decembre 2007, modifie le 18 janvier 2008
From the abstract: "The paper reviews the main (bibliometric) data bases and indicators in use. It demonstrates that these instruments give a biased information about the scientific output of research in Social Sciences and Humanities."

Comment on this paper:
Krichel, T. (2007), bibliometrics and open access solutions, Budapest Open Access Initiative: BOAI Forum Archive, 25 December 2007: "I am somewhat saddened to read that this survey does not discuss CitEc, which I think is the largest open-access citation index in the social sciences."

Added 28 July 2008 Rossner, M., Van Epps, H. and Hill, E. (2007)
Show me the data (editorial)
The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092, published online December 17, 2007
"Just as scientists would not accept the findings in a scientific paper without seeing the primary data, so should they not rely on Thomson Scientific's impact factor, which is based on hidden data. As more publication and citation data become available to the public through services like PubMed, PubMed Central, and Google Scholar®, we hope that people will begin to develop their own metrics for assessing scientific quality rather than rely on an ill-defined and manifestly unscientific number."

Comment on this paper:
Pendlebury, D. A., Thomson Scientific Corrects Inaccuracies In Editorial, Thomson Reuters Citation Impact Forum, undated: "Rossner, Van Epps, and Hill (argue) that Thomson Scientific's impact factor measure for the evaluation of journals should not be trusted since an article data set purchased from Thomson by The Rockefeller University Press did not exactly replicate the Journal Citation Reports data for its own -- and selected other -- journals. When these data were questioned by The Rockefeller University Press, Thomson staff explained precisely the content of the data, as well as its derivation and use. Unfortunately for the readers of the Rossner editorial, the authors misunderstood much and as a result, misled readers about several matters, not only regarding the data but what Thomson representatives did and said from June to September 2007 in many email exchanges."

Added 22 August 2007 Citrome, L. (2007)
Impact Factor? Shmimpact Factor! The Journal Impact Factor, Modern Day Literature Searching, and the Publication Process
Psychiatry, 4(5):54-57, 2007

Added 17 January 2007 Bornmann, L. and Daniel, H.-D. (2007)
What do citation counts measure? A review of studies on citing behavior
Author eprint, undated, Journal of Documentation, accepted for publication

Added 17 January 2007 Meho, L. I. (2006)
The Rise and Rise of Citation Analysis
Author eprint, dLIST, 31 December 2006, Physics World, January 2007
"Provides a historical background of citation analysis, impact factor, new citation data sources (e.g., Google Scholar, Scopus, NASA's Astrophysics Data System Abstract Service, MathSciNet, ScienceDirect, SciFinder Scholar, Scitation/SPIN, and SPIRES-HEP), as well as h-index, g-index, and a-index."

Added 19 November 2006 Electronic Publishing Services and Oppenheim, C. (2006)
UK scholarly journals: 2006 baseline report: An evidence-based analysis of data concerning scholarly journal publishing, see Area 4: Citations, impact factors and their role
Research Information Network, Research Councils UK and the Department of Trade & Industry, October 3, 2006

Comment:
Harnad, S. (2006) Critique of EPS/RIN/RCUK/DTI "Evidence-Based Analysis of Data Concerning Scholarly Journal Publishing", Open Access Archivangelism, October 9. 2006

Added 3 May 2007 Ewing, J. (2006)
Measuring Journals
Notices of the AMS, Vol. 53, No. 9, October 2006, 1049-1053
"in many respects usage statistics are even more flawed than the impact factor, and once again, the essential problem is that there are no explicit principles governing their interpretation. ... while usage statistics are only slightly useful, their misuse can be enormously damaging."

Comment:
Velterop, J. RE: UKSG Usage Factor Research - an Update, liblicense, March 9. 2007: "Ewing further says that "Distrust of 'subjective' scholarly judgment is a modern disease -- one that is profoundly anti-intellectual." I would add that blind trust in 'objective' measurements is equally profoundly anti-intellectual."
Davis, P. RE: UKSG Usage Factor Research - an Update, liblicense, March 10. 2007: "Like citations, usage statistics do not give us an absolute notion of value of journals or articles, yet they do provide us with a measure of utility, and for the sciences, utility is a very powerful measure for how ideas get transmitted through communities and are incorporated into current research. Unlike citations, usage statistics give us a sense of the community of readers (which include authors) and not just the author community. Article downloads provide a robust estimate of the size of user communities, and are also predictive of future citations. In fact, a single week of article downloads from BMJ can predict citations five years later."

Added 8 March 2007 Garfield, E. (2006)
Commentary: Fifty years of citation indexing
International Journal of Epidemiology, 2006 35(5):1127-1128, published online September 19, 2006

Added 03 August 2006 PLoS Medicine Editors (2006)
The Impact Factor Game: It is time to find a better way to assess the scientific literature
PLoS Medicine, Vol. 3, No. 6, June 2006

Added 15 May 2006 Altbach, P. G. (2006)
The Tyranny of Citations
Inside Higher Ed, May 8, 2006

Added 28 February 2006 Noruzi, A. (2006)
The Web Impact Factor: a critical review (pdf, 10pp)
E-LIS, February 9, 2006, in The Electronic Library, 24 (2006)
"Web Impact Factor (WIF) is a quantitative tool for evaluating and ranking web sites ... search engines provide similar possibilities for the investigation of links between web sites/pages to those provided by the academic journals citation databases from the Institute of Scientific Information (ISI). But the content of the Web is not of the same nature and quality as the databases maintained by the ISI."

Added 28 February 2006 Bollen, J., Rodriguez, M. A. and Van de Sompel, H. (2006)
Journal Status (pdf, 16pp)
Arxiv, 9 January 2006
"By merely counting the amount of citations and disregarding the prestige of the citing journals, the ISI IF is a metric of popularity, not of prestige. We demonstrate how a weighted version of the popular PageRank algorithm can be used to obtain a metric that reflects prestige. ... Furthermore, we introduce the Y-factor which is a simple combination of both the ISI IF and the weighted PageRank, and find that the resulting journal rankings correspond well to a general understanding of journal status."

Added 03 August 2006 Moed, H.F. (2005)
Citation analysis of scientific journals and journal impact measures
Current Science, 89 (12): 1990-1996, December 25, 2005

Added 28 February 2006 Dong, P., Loh, M. and Mondry, A. (2005)
The "impact factor" revisited
Biomedical Digital Libraries, December 2005
This is a review, so the findings are not new, but this is perhaps the first such paper to reflect on the effect of free and online availability on journal impact factors, among other IF-related issues.

Added 30 December 2005 Hardy, R., Oppenheim, C., Brody, T. and Hitchcock, S. (2005)
Open Access Citation Information (.doc, 105pp)
Author eprint, November 11, 2005, JISC Committee for the Information Environment (JCIE) Scholarly Communication Group, September 2005
Describes a proposal to increase the exposure of open access materials and their references to indexing services, and to motivate new services by reducing setup costs.

Added 8 March 2007 Perkel, J. M. (2005)
The Future of Citation Analysis
The Scientist, Vol. 19, No. 20, October 24, 2005
"The challenge is to track a work's impact when published in nontraditional forms"

Added 30 December 2005 Monastersky, R. (2005)
Impact Factors Run Into Competition
Chronicle of Higher Education, October 14, 2005

Comment on this article:
Harnad, S. IFs: solution is obvious "Although Richard Monasterky describes a real problem -- the abuse of journal impact factors -- its solution is so obvious -- (a) wealth of powerful new resources are on the way for measuring and analyzing research usage and impact online" American Scientist Open Access Forum, 10 October 2005
Bensman, S. J. Good copy, bad science "I found his article to be unfair, since he concentrated on the shenanigans that are being played with impact factor and supposed errors of ISI in constructing impact factor. This makes for good copy but bad science." Sigmetrics listserv, 18 November 2005
Leydesdorff, L. Discipline-specific impact factor "Monasterky's article lists a number of problems with the ISI-impact factor. However, he fails to mention that the average impact factors vary among fields of science. For example, impact factors in toxicology are considerably lower than in immunology. ... A fix to these problems might be a discipline-specific impact factor. ... Using ISI's Journal Citation Reports, I created the raw materials to make maps of the citation neighborhoods of all the journals." Sigmetrics listserv, 16 September 2005

Added 28 February 2005 Garfield, E. (2005)
The Agony and the Ecstasy - The History and Meaning of the Journal Impact Factor (pdf, 22pp)
International Congress on Peer Review and Biomedical Publication, Chicago, September 16, 2005
Garfield's typically dry, data-filled but essential take on JIFs.

Publishers promote impact factors of OA journals
BioMed Central "Open access journals get impressive impact factors" 23 June 2005
Public Library of Science "The first impact factor for PLoS Biology - 13.9" 27 June 2005
See also this discussion of these announcements on SPARC Open Access Forum, prompted by Elsevier's response from Tony McSean, followed by David Goodman, Charles Bailey, (both 8 July) and Matthew Cockerill (10 July), or see this summary of the discussion: "BMC’s Impact Factors: Elsevier’s Take and Reactions to It", Digital Koans (Charles Bailey's Weblog), 11 July 2005, including Peter Suber's conclusion: "It’s important to distinguish the citation impact of an individual article from a journal impact factor. The BMC-Elsevier debate is about the latter. But OA is more likely to rise and fall according to the former."

Abbasi, K. (2004)
Let's dump impact factors
BMJ, Vol. 329, 16 October 2004
BMJ Rapid Responses to this editorial; also see this list response

Baudoin, L., Haeffner-Cavaillon, N., Pinhas, N., Mouchet, S. and Kordon, C. (2004)
Bibliometric indicators: realities, myth and prospective (abstract only, full paper in French)
Med Sci (Paris), 20 (10):909-15, October 2004

Jacsó, P. (2004)
The Future of Citation Indexing - Interview with Dr. Eugene Garfield (pdf 3pp)
Author eprint, in Online, January 2004

Cockerill, M. J. (2004)
Delayed impact: ISI's citation tracking choices are keeping scientists in the dark
BMC Bioinformatics 2004, 5:93, 12 July 2004

Added 26 September 2005 Shin E. J. (2003)
Do Impact Factors change with a change of medium? A comparison of Impact Factors when publication is by paper and through parallel publishing (abstract only)
Journal of Information Science, 29 (6): 527-533, 2003
"it is found that Impact Factors of (journals from the period) 2000 and 2001 were significantly higher than those of 1994 and 1995 in the journals published by parallel publishing (combination journals–simultaneous publication of paper and electronic journals). In particular, the Impact Factors of the combination journals increased after the journals transformed their available media from paper journals to combination ones."

Walter, G., Bloch, S., Hunt, G. and Fisher, K. (2003)
Counting on citations: a flawed way to measure quality
MJA, 2003, 178 (6): 280-281

Borgman, C. L. and Furner, J. (2002)
Scholarly Communication and Bibliometrics, author preprint (pdf 45pp)
Author eprint, in Annual Review of Information Science and Technology, Vol. 36, edited by B. Cronin, 2002

Guédon, J.-C. (2001)
In Oldenburg’s Long Shadow: Librarians, Research Scientists, Publishers, and the Control of Scientific Publishing
Creating the Digital Future, Proceedings of the 138th Annual Meeting, Association of Research Libraries, Toronto, Ontario, May 23-25, 2001

Garfield, E. (1999)
Journal impact factor: a brief review
CMAJ, 161 (8), October 19, 1999

Wouters, P. (1999)
The Citation Culture (pdf 290pp)
PhD Thesis, University of Amsterdam, 1999

Garfield, E. (1998)
The use of journal impact factors and citation analysis in the evaluation of science
Author eprint, presented at the 41st Annual Meeting of the Council of Biology Editors, Salt Lake City, UT, May 4, 1998

Seglen, P. O. (1997)
Why the impact factor of journals should not be used for evaluating research
BMJ, 314:497, 15 February 1997

Garfield, E. (1973)
Citation Frequency as a Measure of Research Activity and Performance (pdf 3pp)
Author eprint, in Essays of an Information Scientist, 1: 406-408, 1962-73, Current Contents, 5, January 31, 1973

Garfield, E. (1955)
Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas
Author eprint, in Science, Vol:122, No:3159, p.108-111, July 15, 1955

Open access

Notes. In printed form, little of the published research literature was free. With more material beginning to appear on the Web from the mid-1990s, more became freely available. Open access is in a sense a formalisation of that process, a recognition that all published, refereed scholarly papers could and should be freely accessible in some form to everyone online without compromising the quality and integrity of the literature. That is the goal. This simple idea, especially when focussed on this very specific literature, seems to have been quite difficult to grasp for many bound by the old, pre-online ways of thinking. Despite the often antithetical tone of the debate, progress has been rapid since the landmark of the Budapest Open Access Initiative in February 2002, even impinging on prospective government policies by 2003 (e.g. Martin Sabo's Public Access to Science Act; UK House committee releases its report on open access; Major development in providing OA to taxpayer-funded research). It has all been brilliantly logged by Peter Suber in Open Access News (http://www.earlham.edu/~peters/fos/fosblog.html), but for a very quick overview the following papers are sufficient.

Suber, P. (updated)
Open Access Overview

Swan, A. (2007)
Open Access and the Progress of Science
American Scientist, April-June 2007
Swan justified open access in support of her 'progress' article in a list discussion. See blogged extracts from that discussion.

Swan, A. (2006)
Open Access: Why should we have it?
presented at "Zichtbaar onderzoek. Kan Open Archives daarbij helpen?" / Visible research. Can OAI help? Organised by AWI (Flemish Ministry for Economy, Enterprise, Science, Innovation and Foreign Trade) and VOWB (Flemish Organisation of Scientific Research Libraries), May 2006

Swan, A. (2005)
Open Access
JISC, Briefing Paper, 1 April 2005

Suber, P. (2004)
A Primer on Open Access to Science and Scholarship
Author eprint, in Against the Grain, Vol. 16, No. 3, June 2004

Harnad, S. (2004)
The Green Road to Open Access: A Leveraged Transition
American Scientist Forum, January 07, 2004

Suber, P. (2003)
Removing the Barriers to Research: An Introduction to Open Access for Librarians
Author eprint, in College & Research Libraries News, 64, February, 92-94, 113

^Top
<Home

The OpCit Project
This page produced and maintained by the Open Citation project. Contact us