Delayed impact: ISI's citation tracking choices are keeping scientists in the dark

Impact factors – love them or loathe them, as a publisher it is difficult to ignore them. The number one question that BioMed Central gets asked by potential authors is "What is the impact factor of the journal XXXXXX?".

The scientific community has come to regard impact factors, calculated by the Institute for Scientific Information (ISI), as providing a quantitative and largely objective guide to which journals publish the best research.

Although many problems can result from naïve reliance on journal impact factors as a quality metric (especially when attempting to compare different fields) [1], the perception of many scientists is that, to get recognition and career advancement, they must publish in a journal with a good impact factor.

The problem

An impact factor is defined by a simple calculation: "The impact factor is calculated by dividing the number of current citations to articles published in the two previous years by the total number of articles published in the two previous years."[2].

This presents a major obstacle to publishers of new journals, since even the best journal won't have a proper impact factor for three years. But this situation is even worse than it sounds, as the clock only starts ticking when ISI starts "tracking" the journal.

When does ISI start tracking the journal? It depends… According to ISI, the factors it uses to decide when to start tracking a journal include:

 ·  how many articles the journal publishes

 ·  how many competing journals ISI already tracks in the same discipline

 ·  the previous citation record of the journal's editorial board

 ·  the previous citation record of the authors who publish in the journal

 ·  the number of times the journal has been cited in journals that are already tracked by ISI

Unfortunately, despite good intentions, this selectivity on ISI's part has the unintended consequence of concealing the success of new journals. A case in point is BMC Bioinformatics. Since it published its first article in 2000, this journal has rapidly established itself as one of the most active and successful in its field (see Figure 1).

However, since ISI only began tracking BMC Bioinformatics in 2002, this will not translate into an official impact factor until June 2005, when the 2004 impact factors are released.

All is not lost, however. Johannes Stegmann noted in correspondence to Nature [3] that it is possible to calculate an unofficial impact factor for any journal, even if it is not officially tracked by ISI, by making use of the information in ISI's cited reference database which includes the entire reference list of all tracked journal articles, and therefore includes citations of journals which are not themselves tracked. Using this method, the 2003 impact factor for BMC Bioinformatics can be estimated as follows:

      BMC Bioinformatics articles published in 2001-2

      48   (one 'Correction' article ignored for citation analysis purposes)

      2003 citations of these BMC Bioinformatics articles

      235   (according to ISI Web of Science cited reference database)

      Unofficial 2003 impact factor for BMC Bioinformatics

      Impact factor = 235/48 = 4.896

As Table 1 shows, this "unofficial" 2003 impact factor for BMC Bioinformatics already compares very favourably with that of more established journals.

This estimated impact factor places BMC Bioinformatics in the top 5% of all journals covered by ISI. Yet an author reviewing the 2003 Journal Citation Report from ISI would have no idea that the journal BMC Bioinformatics was so highly cited.

The number of articles on which this calculation is based is relatively small. The official impact factor, which is expected to arrive in mid-2005, may be significantly higher or lower, but it seems clear that it will be respectable.

BMC Bioinformatics is not alone in facing this problem: there are many other recently launched journals, both from BioMed Central and from other publishers, whose impressive citation record is not currently captured by the impact factors listed in ISI's Journal Citation Report. It is an unfortunate fact that this may needlessly dissuade many authors from publishing in these new journals, and thus may serve to hold back innovation in science publishing.

Figure 1

The chart shows the number of articles published in BMC Bioinformatics, and the number of times the journal has been cited, over the time span 2001-3.

Table 1 A comparison of 2003 journal impact factors

What can be done?

After many years of having the field of citation analysis largely to itself, ISI is finally facing the prospect of serious competition. The increasing use of standard XML formats by publishers mean that citation analysis is no longer a daunting logistical challenge. It is simply a question of number crunching.

Citation tracking data for Open Access content is already available through Citebase [4], and the usefulness of this free service will grow as the amount of Open Access content increases. Meanwhile, CrossRef (the full text linking service) is also now collecting article reference lists from publishers for 'forward linking' purposes [5], and these could in future potentially also be used to calculate impact-factor-like metrics. In addition Elsevier is now working on Scopus [6], a bibliographic database/citation analysis service that the publisher claims will offer broader journal coverage than ISI.

With luck, this competition may give ISI just the spur it needs. BioMed Central's recommendation is that ISI should reconsider its policy on citation tracking, and should introduce a policy of immediately tracking any peer-reviewed journal that meets basic quality standards and which can provide reference list data in an appropriate form to allow automated analysis. By doing this, ISI would provide a valuable impartial service to the scientific community.