Home | You are at

Citation Validation

Last updated September 07 2000 11:00:57.

The output from Spotcites

Comparing "orange" and "red" links with the total number of citations. For the opcit project certain classes of identifiable citations have been colour coded (for development purposes):

RedA author has specifically cited a paper in arXiv, e.g. "J. Smith et al, preprint hep-th/9909999").
OrangeA author has cited a paper using its publication information (journal, issue, page) which has been deposited in the archive and found using the journal reference in arXiv.

Using the output from Spotcites red links can be identified by lines that contain a arXiv reference more than once. A table can be built of paper * cited papers * orange|red. This table can then be used to build the number of orange/red links for a given month.

awk '{ print $1,$3 }' ZZZ.type | ../arxiv2date | awk '{ print $3,$2 }' | grep R$ | ../countcol -v -n
awk '{ print $1,$3 }' ZZZ.type | ../arxiv2date | awk '{ print $3,$2 }' | grep O$ | ../countcol -v -n

This graph shows how many citations have been identified, retrospectively, for papers deposited each month. The total number of "lines of citations" is shown in blue (not the total of red and orange), i.e. the background population from which citations are being identified from. The blue line corresponds to the right-hand y-axis and is scaled down.

Excel Spreadsheet Source

The numbers of red and orange citations found can be compared against the actual number of citations in the following graph. The green line corresponds to the sum of red and orange links scaled for comparison to the total number of citations.

The total number of citations can be factored out by dividing the red, orange and sum by the total citations for the equivalent month. This shows how increasingly spotcites is more effective at identifying citations, and that more citations are to papers in arXiv.

A comparison can be made between red and orange links, i.e. the changing behaviour of authors from citing post-print articles to citing e-print articles. The following graph shows how the number of direct citations has increased at a greater rate than the number of identifiable citations to published articles.

The graphs so far have covered the entire archive's deposits. It would be useful to identify how well citations are being identified within certain subject areas.


awk '{ print $1 }' SCOOT.HEPTH | ../arxiv2date | awk '{ print $2 }' | ../countcol -v -n
awk '{ print $1,$3 }' ZZZ.type | grep ^hep-th | ../arxiv2date | awk '{ print $3,$2 }' | grep R$ | ../countcol -v -n
awk '{ print $1,$3 }' ZZZ.type | grep ^hep-th | ../arxiv2date | awk '{ print $3,$2 }' | grep O$ | ../countcol -v -n


Why are 40% of citations red-links - citations to LANL e-prints?



In addition to analysing citation identification we can also add the ratio of citations to "antique" papers, citations to papers that were published before the start of the archive. As would be expected the percentage of citations to antique papers reduces over time (i.e. currently a paper would have to be more than 10 years old to be antique").

When comparing the trend of antique citations to the increasing percentage of successfully identified red and orange citations we can see that citations to papers that are potentially not in the archive has remained relatively constant (from 50% to 30%), in contrast to the increase of identifiable citations to papers in the archive (from 0% to 50%). If we assume that spotcites is capable of identifying 100% of citations to articles in arXiv this suggests that the ratio of citations to articles not in arXiv has remained relatively constant - this could be because:

  1. these are citations to non-arXiv media (e.g. books)
  2. the cited author has decided not to deposit their articles in arXiv
  3. the numbers of articles outside of arXiv are constantly being added to because antiques are reducing

Negative Latency Citations

There are 6455 instances of citations having a negative latency (that is that a paper cites another paper before the cited paper is deposited in the archive). These are from 4259 papers.

This number can be seperated into those that are citations to LANL preprints (3479) and those that are citations to published articles (2976), but the published article has been deposited in arXiv. Negative latency citations to LANL articles can only occur if an author updates the paper, and adds additional references. The total number of papers that this has occured in is 2095 (1.58% of 132219 papers).

Checking a sample of citations

A sample of 100 citations (the last 100) were taken from papers deposited in 1999, 1997 and 1995. For each citation the relevent line was found from the output of spotcites (the source citation text) and compared to the abstract of the cited paper. Primarily the author's names were used for verification, as spotcites is based on matching publication data.

Paper References Type Correct  
hep-th/9912286 hep-th/9808140 O 1  
hep-th/9912286 hep-th/9901023 R 1  
hep-th/9912286 hep-th/9707163 O 1  
hep-th/9912286 hep-th/9802074 O 1  
hep-th/9912286 hep-th/9707163 O 1  
hep-th/9912286 hep-th/9810068 O 1  
hep-th/9912286 hep-ph/9501213 O 1  
hep-th/9912286 hep-th/9806074 O 1  
hep-th/9912287 hep-ph/9905221 O 1  
hep-th/9912287 hep-th/9906064 O 1  
hep-th/9912287 hep-th/9910093 O 1  
hep-th/9912287 gr-qc/9910076 R 1  
hep-th/9912287 hep-th/9911055 R 1  
hep-th/9912287 hep-ph/9906513 R 1  
hep-th/9912287 hep-ph/9911406 R 1  
hep-th/9912287 hep-ph/9912266 R 1  
hep-th/9912287 hep-th/9911043 R 1  
hep-th/9912287 hep-th/9910149 O 1  
hep-th/9912287 gr-qc/9912002 R 1  
hep-th/9912287 hep-th/9910219 R 1  
hep-th/9912287 hep-th/9911165 R 1  
hep-th/9912287 hep-th/9911181 R 1  
hep-th/9912287 hep-th/9905210 R 1  
hep-th/9912287 hep-ph/9905487 R 1  
hep-th/9912287 hep-th/9909053 R 1  
hep-th/9912287 hep-th/9905012 R 1  
hep-th/9912287 hep-ph/9910498 R 1  
hep-th/9912287 hep-ph/9906523 R 1  
hep-th/9912287 hep-ph/9911406 R 1  
hep-th/9912287 hep-ph/9909481 R 1  
hep-th/9912287 hep-th/9912118 R 1  
hep-th/9912287 hep-th/9912001 R 1  
hep-th/9912287 hep-th/9905111 R 1  
hep-th/9912287 hep-th/9911230 R 1  
hep-th/9912287 hep-th/9912233 R 1  
hep-th/9912288 hep-th/9711162 R 1  
hep-th/9912288 hep-th/9711165 R 1  
hep-th/9912288 hep-th/9810179 R 1  
hep-th/9912288 hep-th/9907166 R 1  
hep-th/9912288 hep-th/9908134 R 1  
hep-th/9912288 hep-th/9908142 R 1  
hep-th/9912288 hep-th/9908056 R 1  
hep-th/9912288 hep-th/9711200 R 1  
hep-th/9912288 hep-th/9911057 R 1  
hep-th/9912288 hep-th/9911031 R 1  
hep-th/9912288 hep-th/9910083 R 1  
hep-th/9912288 hep-th/9910092 R 1  
hep-th/9912288 hep-th/9703040 R 1  
hep-th/9912288 hep-th/9708005 R 1  
hep-th/9912288 hep-th/9802109 R 1  
hep-th/9912288 hep-th/9805140 R 1  
hep-th/9912288 hep-th/9803023 R 1  
hep-th/9912288 hep-th/9804095 R 1  
hep-th/9912288 hep-th/9805050 R 1  
hep-th/9912288 hep-th/9909059 R 1  
hep-th/9912288 hep-th/9911193 R 1  
hep-th/9912288 hep-th/9911183 R 1  
hep-th/9912288 hep-th/9911137 R 1  
hep-th/9912288 hep-th/9907024 R 1  
hep-th/9912288 hep-th/9908092 R 1  
hep-th/9912288 hep-th/9404071 R 1  
hep-th/9912290 hep-th/9808042 R 1  
hep-th/9912290 hep-th/9810022 R 1  
hep-th/9912290 hep-th/9912092 R 1  
hep-th/9912290 hep-th/0003188 R 1  
hep-th/9912290 q-alg/9707029 R 1  
hep-th/9912290 hep-th/9901099 R 1  
hep-th/9912290 hep-th/9810087 R 1  
hep-th/9912290 hep-th/9904044 R 1  
hep-th/9912290 hep-th/9909185 R- 0  
hep-th/9912290 hep-ph/9504352 R 1  
hep-th/9912290 hep-th/9609128 R 1  
hep-th/9912290 hep-th/9607174 R 1  
hep-th/9912290 err-qa/9904055 R 0  
hep-th/9912290 hep-th/9604128 R- 0  
hep-th/9912290 hep-th/9903249 R 1  
hep-th/9912291 hep-th/9802149 R 1  
hep-th/9912291 hep-th/9908149 R 1  
hep-th/9912291 quant-ph/9909084 R 1  
hep-th/9912292 hep-th/9406217 O 1  
hep-th/9912292 hep-th/9608177 O 1  
hep-th/9912292 hep-th/9707061 O 1  
hep-th/9912292 hep-th/9711203 O 1  
hep-th/9912292 hep-th/9811232 O 1  
hep-th/9912292 hep-th/9811130 O 1  
hep-th/9912292 hep-th/9811244 O 1  
hep-th/9912292 hep-th/9901003 O 1  
hep-th/9912292 hep-th/9606139 O 1  
hep-th/9912292 hep-th/9702202 O 1  
hep-th/9912292 hep-th/9702163 O 1  
hep-th/9912292 hep-th/9710129 O 1  
hep-th/9912292 hep-th/9811022 O 1  
hep-th/9912292 hep-th/9904109 R 1  
hep-th/9912292 hep-th/9909098 R 1  
hep-th/9912292 hep-th/9711116 O 1  
hep-th/9912293 hep-th/9908142 R 1  
hep-th/9912293 hep-th/9908105 R 1  
hep-th/9912293 hep-th/9909132 R 1  
hep-th/9912293 hep-th/9909081 R 1  
hep-th/9912293 hep-th/9907211 R 1  
hep-th/9712218 hep-th/9607076 O 1  
hep-th/9712218 hep-th/9409037 O 1  
hep-th/9712218 hep-th/9507003 O 1  
hep-th/9712218 hep-th/9708036 R 1  
hep-th/9712218 hep-th/9601162 O 1  
hep-th/9712218 hep-th/9506102 O 1  
hep-th/9712224 hep-th/9606033 O 1  
hep-th/9712224 hep-th/9601029 O 1  
hep-th/9712224 hep-th/9602051 O 1  
hep-th/9712224 hep-th/9609026 O 1  
hep-th/9712224 hep-th/9604166 O 1  
hep-th/9712224 hep-th/9604179 O 1  
hep-th/9712224 hep-th/9612095 O 1  
hep-th/9712224 gr-qc/9409013 O 1  
hep-th/9712224 hep-th/9407118 O 1  
hep-th/9712224 hep-th/9509093 O 1  
hep-th/9712224 gr-qc/9601032 O 1  
hep-th/9712224 hep-th/9611095 R 1  
hep-th/9712230 hep-th/9603167 R 1  
hep-th/9712230 hep-th/9608024 R 1  
hep-th/9712230 hep-th/9606165 R 1  
hep-th/9712230 hep-th/9610140 R 1  
hep-th/9712230 hep-th/9704151 O 1  
hep-th/9712230 hep-th/9610043 R 1  
hep-th/9712230 hep-th/9612126 R 1  
hep-th/9712230 hep-th/9702203 O 1  
hep-th/9712230 hep-th/9707019 R 1  
hep-th/9712230 hep-th/9710178 R 1  
hep-th/9712230 hep-th/9709220 R 1  
hep-th/9712230 hep-th/9710009 R 1  
hep-th/9712230 hep-th/9611214 R 1  
hep-th/9712230 hep-th/9611137 R 1  
hep-th/9712230 hep-th/9603127 R 1  
hep-th/9712230 hep-th/9702203 ? ?  
hep-th/9712230 hep-th/9609017 R 1  
hep-th/9712230 hep-th/9707093 R 1  
hep-th/9712230 hep-th/9712049 R 1  
hep-th/9712236 hep-th/9711005 R 1  
hep-th/9712236 hep-th/9610043 R 1  
hep-th/9712236 hep-th/9608137 R 1  
hep-th/9712236 hep-th/9710217 R 1  
hep-th/9712236 hep-th/9710226 R 1  
hep-th/9712236 hep-th/9712202 R 1  
hep-th/9712236 hep-th/9709091 R 1  
hep-th/9712236 hep-th/9709108 R 1  
hep-th/9712236 hep-th/9711003 R 1  
hep-th/9712236 hep-th/9704080 R 1  
hep-th/9712236 hep-th/9710009 O 1  
hep-th/9712236 hep-th/9709220 R 1  
hep-th/9712236 hep-th/9712063 R 1  
hep-th/9712236 hep-th/9706072 R 1  
hep-th/9712242 hep-th/9509074 O 1  
hep-th/9712242 hep-th/9210021 O 1  
hep-th/9712242 hep-th/9403060 O 1  
hep-th/9712242 hep-th/9506027 O 1  
hep-th/9712242 hep-th/9510117 O 1  
hep-th/9712242 hep-th/9602087 O 1  
hep-th/9712242 hep-th/9211021 O 1  
hep-th/9712248 hep-lat/9607068 O 1  
hep-th/9712248 hep-lat/9709042 R 1  
hep-th/9712248 hep-lat/9711009 R 1  
hep-th/9712248 hep-lat/9610005 O 1  
hep-th/9712248 hep-lat/9708023 R- 1  
hep-th/9712248 hep-lat/9710039 R 1  
hep-th/9712248 hep-th/9402066 O 1  
hep-th/9712248 hep-th/9511015 O 1  
hep-th/9712248 hep-th/9710128 R 1  
hep-th/9712248 hep-th/9511104 O 1  
hep-th/9712248 hep-th/9605116 O 1  
hep-th/9712254 hep-th/9608082 R 1  
hep-th/9712254 hep-th/9111036 O 1  
hep-th/9712254 hep-th/9206002 O 1  
hep-th/9712257 hep-th/9611064 O 1  
hep-th/9712257 cond-mat/9612187 O 1  
hep-th/9712257 hep-th/9510017 O 1  
hep-th/9712257 hep-th/9707068 R 1  
hep-th/9712257 hep-th/9711029 R 1  
hep-th/9712257 hep-th/9605042 O 1  
hep-th/9712257 hep-th/9311183 O 1  
hep-th/9712257 hep-th/9503207 O 1  
hep-th/9712257 hep-th/9506014 O 1  
hep-th/9712257 hep-th/9607229 O 1  
hep-th/9712257 hep-th/9702037 O 1  
hep-th/9712257 hep-th/9701190 O 1  
hep-th/9712257 hep-th/9712186 R 1  
hep-th/9712257 hep-th/9610066 O 1  
hep-th/9712257 hep-th/9309105 O 1  
hep-th/9712257 hep-ph/9509246 O 0  
hep-th/9712257 hep-th/9708141 O 1  
hep-th/9712257 hep-th/9702195 R 1  
hep-th/9712257 hep-th/9506135 O 1  
hep-th/9712257 hep-th/9612148 O 1  
hep-th/9712257 hep-th/9708166 R 1  
hep-th/9712257 hep-th/9610192 O 1  
hep-th/9712257 hep-th/9711201 R 1  
hep-th/9712257 hep-th/9606112 O 1  
hep-th/9712257 hep-th/9509171 O 1  
hep-th/9712257 hep-th/9706057 R 1  
hep-th/9712257 hep-th/9711069 R 1  
hep-th/9712257 hep-th/9708040 R 1  
hep-th/9512173 hep-th/9303057 O 1  
hep-th/9512173 hep-th/9304154 O 1  
hep-th/9512173 astro-ph/9402057 O 1  
hep-th/9512173 hep-th/9408099 O 1  
hep-th/9512173 hep-th/9408155 O 1  
hep-th/9512173 hep-th/9411149 O 1  
hep-th/9512173 hep-th/9408074 O 1  
hep-th/9512173 hep-th/9411102 O 1  
hep-th/9512173 hep-th/9502013 O 1  
hep-th/9512173 hep-th/9502057 O 1 (conf. Paper refernced in citation)
hep-th/9512173 hep-th/9412200 R 1  
hep-th/9512173 hep-th/9503121 O 1  
hep-th/9512173 hep-th/9503234 R 1  
hep-th/9512173 hep-th/9505004 O 1  
hep-th/9512173 hep-th/9505006 O 1  
hep-th/9512173 hep-th/9505051 O 1  
hep-th/9512173 hep-th/9505135 O 1  
hep-th/9512173 hep-th/9505186 R 1  
hep-th/9512173 hep-th/9506035 R 1  
hep-th/9512173 hep-th/9506148 R 1  
hep-th/9512173 hep-th/9507040 R 1  
hep-th/9512173 hep-th/9507112 R 1  
hep-th/9512173 hep-th/9508021 R 1  
hep-th/9512173 hep-th/9508091 R 0  
hep-th/9512173 hep-th/9508140 R 1  
hep-th/9512179 hep-ph/9305224 O 1  
hep-th/9512179 hep-th/9212009 O 1  
hep-th/9512179 hep-ph/9310330 O 1  
hep-th/9512179 hep-th/9409019 O 1  
hep-th/9512179 hep-th/9405176 O 1  
hep-th/9512179 hep-th/9403038 O 1  
hep-th/9512179 hep-th/9509020 R 1  
hep-th/9512179 hep-th/9509027 R 1  
hep-th/9512179 hep-th/9307146 O 1  
hep-th/9512179 hep-th/9504026 R 1  
hep-th/9512179 hep-ph/9502214 O 1  
hep-th/9512179 hep-ph/9507450 R 1  
hep-th/9512179 hep-th/9302119 O 1  
hep-th/9512179 hep-th/9311108 O 1  
hep-th/9512179 hep-th/9309128 O 1  
hep-th/9512179 hep-th/9402015 O 1  
hep-th/9512179 hep-th/9401153 O 1  
hep-th/9512185 hep-ph/9403224 O 1  
hep-th/9512185 hep-th/9406081 O 1  
hep-th/9512191 hep-th/9511030 R 1  
hep-th/9512191 hep-th/9510017 R 1  
hep-th/9512191 hep-th/9304104 O 1  
hep-th/9512191 hep-th/9512053 R 1  
hep-th/9512197 hep-th/9407027 O 1  
hep-th/9512197 hep-th/9505084 R 1  
hep-th/9512203 hep-th/9412184 O 1  
hep-th/9512203 hep-th/9410167 O 1  
hep-th/9512203 hep-th/9501068 O 1  
hep-th/9512203 hep-th/9504095 O 1  
hep-th/9512203 hep-th/9507048 R 1  
hep-th/9512203 hep-th/9506126 R 1  
hep-th/9512203 hep-th/9507158 R 1  
hep-th/9512203 hep-th/9510086 R 1  
hep-th/9512203 hep-th/9508154 R 1  
hep-th/9512203 hep-th/9511159 R 1  
hep-th/9512203 hep-th/9511213 R 1  
hep-th/9512203 hep-th/9509070 R 1  
hep-th/9512203 hep-th/9510209 R 1  
hep-th/9512203 hep-th/9512181 R 1  
hep-th/9512203 hep-th/9512196 R 1  
hep-th/9512203 hep-th/9510017 R 1  
hep-th/9512203 hep-th/9510135 R 1  
hep-th/9512203 hep-th/9511030 R 1  
hep-th/9512203 hep-th/9510161 R 1  
hep-th/9512203 hep-th/9511222 R 1 (two redlinks, only one picked up)
hep-th/9512203 hep-th/9510169 R 1  
hep-th/9512203 hep-th/9511026 R 1  
hep-th/9512203 hep-th/9512078 R 1  
hep-th/9512203 hep-th/9511043 R 1  
hep-th/9512203 hep-th/9511173 R 1  
hep-th/9512203 hep-th/9511194 R 1  
hep-th/9512203 hep-th/9512059 R 1  
hep-th/9512203 hep-th/9512062 R 1  
hep-th/9512203 hep-th/9512061 R 1  
hep-th/9512203 hep-th/9512077 R 1  
hep-th/9512203 hep-th/9305185 O 1  
hep-th/9512203 hep-th/9407021 O 1  
hep-th/9512203 hep-th/9501030 R 1  
hep-th/9512203 hep-th/9503124 O 1  
hep-th/9512203 hep-th/9504006 R 1  
hep-th/9512203 hep-th/9504034 R 1  
hep-th/9512206 hep-th/9401070 O 1  
hep-th/9512206 hep-th/9408098 O 1  
hep-th/9512206 hep-th/9410215 O 1  
hep-th/9512206 hep-th/9304037 O 1  
hep-th/9512206 hep-th/9312107 O 1  
hep-th/9512206 hep-th/9506138 R 1  
hep-th/9512212 hep-th/9410137 O 1  
hep-th/9512218 hep-th/9507118 R 1  
hep-th/9512218 hep-th/9507070 R 1  
hep-th/9512218 hep-th/9508080 R 1  
hep-th/9512218 hep-th/9510095 R 1  
hep-th/9512218 hep-th/9511008 R 0  
hep-th/9512218 hep-th/9511081 R 0  
hep-th/9512218 hep-th/9511090 R 0