Delayed impact: ISI's citation tracking choices are keeping scientists in the dark
© Cockerill; licensee BioMed Central Ltd. 2004
Received: 8 July 2004
Accepted: 12 July 2004
Published: 12 July 2004
Impact factors – love them or loathe them, as a publisher it is difficult to ignore them. The number one question that BioMed Central gets asked by potential authors is "What is the impact factor of the journal XXXXXX?".
The scientific community has come to regard impact factors, calculated by the Institute for Scientific Information (ISI), as providing a quantitative and largely objective guide to which journals publish the best research.
Although many problems can result from naïve reliance on journal impact factors as a quality metric (especially when attempting to compare different fields) , the perception of many scientists is that, to get recognition and career advancement, they must publish in a journal with a good impact factor.
An impact factor is defined by a simple calculation: "The impact factor is calculated by dividing the number of current citations to articles published in the two previous years by the total number of articles published in the two previous years.".
This presents a major obstacle to publishers of new journals, since even the best journal won't have a proper impact factor for three years. But this situation is even worse than it sounds, as the clock only starts ticking when ISI starts "tracking" the journal.
When does ISI start tracking the journal? It depends… According to ISI, the factors it uses to decide when to start tracking a journal include:
· how many articles the journal publishes
· how many competing journals ISI already tracks in the same discipline
· the previous citation record of the journal's editorial board
· the previous citation record of the authors who publish in the journal
· the number of times the journal has been cited in journals that are already tracked by ISI
Unfortunately, despite good intentions, this selectivity on ISI's part has the unintended consequence of concealing the success of new journals. A case in point is BMC Bioinformatics. Since it published its first article in 2000, this journal has rapidly established itself as one of the most active and successful in its field (see Figure 1).
However, since ISI only began tracking BMC Bioinformatics in 2002, this will not translate into an official impact factor until June 2005, when the 2004 impact factors are released.
All is not lost, however. Johannes Stegmann noted in correspondence to Nature  that it is possible to calculate an unofficial impact factor for any journal, even if it is not officially tracked by ISI, by making use of the information in ISI's cited reference database which includes the entire reference list of all tracked journal articles, and therefore includes citations of journals which are not themselves tracked. Using this method, the 2003 impact factor for BMC Bioinformatics can be estimated as follows:
BMC Bioinformatics articles published in 2001-2
48 (one 'Correction' article ignored for citation analysis purposes)
2003 citations of these BMC Bioinformatics articles
235 (according to ISI Web of Science cited reference database)
Unofficial 2003 impact factor for BMC Bioinformatics
Impact factor = 235/48 = 4.896
As Table 1 shows, this "unofficial" 2003 impact factor for BMC Bioinformatics already compares very favourably with that of more established journals.
This estimated impact factor places BMC Bioinformatics in the top 5% of all journals covered by ISI. Yet an author reviewing the 2003 Journal Citation Report from ISI would have no idea that the journal BMC Bioinformatics was so highly cited.
The number of articles on which this calculation is based is relatively small. The official impact factor, which is expected to arrive in mid-2005, may be significantly higher or lower, but it seems clear that it will be respectable.
A comparison of 2003 journal impact factors
2003 impact factor
Nucleic Acids Research
Molecular Biology and Evolution
Journal of Computational Biology
Journal of Molecular Evolution
Molecular Phylogenetics and Evolution
What can be done?
After many years of having the field of citation analysis largely to itself, ISI is finally facing the prospect of serious competition. The increasing use of standard XML formats by publishers mean that citation analysis is no longer a daunting logistical challenge. It is simply a question of number crunching.
Citation tracking data for Open Access content is already available through Citebase , and the usefulness of this free service will grow as the amount of Open Access content increases. Meanwhile, CrossRef (the full text linking service) is also now collecting article reference lists from publishers for 'forward linking' purposes , and these could in future potentially also be used to calculate impact-factor-like metrics. In addition Elsevier is now working on Scopus , a bibliographic database/citation analysis service that the publisher claims will offer broader journal coverage than ISI.
With luck, this competition may give ISI just the spur it needs. BioMed Central's recommendation is that ISI should reconsider its policy on citation tracking, and should introduce a policy of immediately tracking any peer-reviewed journal that meets basic quality standards and which can provide reference list data in an appropriate form to allow automated analysis. By doing this, ISI would provide a valuable impartial service to the scientific community.
- Garfield E: Essays/papers on impact factor.[http://www.garfield.library.upenn.edu/impactfactor.html]
- ISI Journal Citation Reports website[http://jcrweb.com/www/help/hjcrgls2.htm]
- Stegmann J: How to evaluate journal impact factors. Nature 1997, 390: 550. 10.1038/37463View ArticlePubMedGoogle Scholar
- CrossRef forward linking service[http://www.crossref.org/01company/pr/press20040608.html]
This article is published under license to BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.