Journal and journal services

In 2009, editors of the journal GENETICS [7] approached WormBase to provide linking from C. elegans articles to WormBase. GENETICS articles are processed by Dartmouth Journal Services (DJS). DJS provides XML formatted files for linking through a web service to WormBase. After linked files undergo a manual quality control step at WormBase, the linked file is returned to DJS. DJS processes the linked file creating a linked PDF, which is sent back to WormBase for final approval and then to the author for a final proof. If mistakes are found during the PDF proof stage they are fixed by DJS. Questions concerning ambiguities and formatting errors that arise during the quality control step at the database are relayed to the author through DJS.

Authors

WormBase provides a web form for the authors (Figure 1) to declare entities they have discovered or described, which do not yet exist in the database. A link to this web form is emailed to authors by the journal as soon as their article is accepted for publication. The form itself should take authors only a matter of minutes to understand and complete. Data entered through the form are added directly to the database-specific lexicon for linking, and thus needs to be machine readable; therefore, data submitted through the form is monitored by a WormBase curator to make sure the data are formatted correctly for the pipeline.

If any new objects declared through the author form are not in WormBase at the time of linking, the links for the new, author-declared entities are silent when they are inserted into the article (prior to actual publication). The author form also alerts WormBase curators to the presence of these new entities, which will then be added to the next release of the database by the curator. Links to these new entities are usually live when the article is published or soon thereafter.

Databases

For S. cerevisiae yeast articles, SGD provides gene, protein, and variation entity lists. The URL pattern used is http://www.yeastgenome.org/cgi-bin/locus.fpl?dbid=SGDID, where SGDID is the SGD identification for the entity.

For D. melanogaster articles, FlyBase provides entities from five different classes: genes, alleles, aberrations, transgenic transposons and transposon insertions. The URL pattern used is http://flybase.org/reports/FBID.html, where FBID is the FlyBase identification for the entity. FlyBase suffers from the unique problem of having several common English words (for example: for, we, a) as gene names. Since we receive XML formatted files from the journal publisher, the policy is to link an entity only if it is italicized i.e. within italic tags in the XML.

Database curators

After the paper has gone through the automated linking script, a link to the marked up paper is sent to the database curator for a manual quality control (QC) step to ensure all appropriate links are made. QC curators use an html viewer to manually assess each link for accuracy. The curator also searches the paper to make sure any new entity has been identified and linked. Additions, deletions and any corrections to the links in the XML file are done using a text editor or an XML editor.

Ensuring stable URLs

An important requirement for this project is long-term stability of the database URLs. The databases need to be dedicated in maintaining the resource pages for the long term. To help the databases check whether the URLs inserted in the articles are live, a table of all the database links is maintained for each article, after all the manual QC steps are completed. Database curators periodically check the URLs to make sure the links are live.

Necessity and benefits of manual quality control

As mentioned above, the manual QC step can be used to deal with new objects that do not exist in WormBase. Whether or not the author declares these entities, we can identify them and incorporate them into the database within the next database release or soon thereafter. Our author first pass form also serves the dual purpose of alerting WormBase curators that the paper being processed contains a data type relevant to their curation pipeline.

In addition to identifying new entities missed by authors, we use the manual QC step to resolve ambiguous links as well as to correct linking errors incurred by author typos or XML file processing. Since we use an automated linking script that identifies entities in a paper based on a list of known entities, if an entity name is used more than once for distinct objects in the database, the script will potentially link the entity name to the wrong page; for example, the entity AB1, in the C. elegans literature, is both a clone and a strain name. We have found and corrected ambiguous links in 15 papers so far via the manual QC step.

Errors from ambiguous entity names can be accurately resolved only through human oversight. For C. elegans articles, little effort is required for resolving ambiguities because the number of ambiguous entities within the C. elegans field is low; this aspect of C. elegans research is a benefit of the simple and strict nomenclature rules that were put in place at the onset of adopting C. elegans as a model organism for genetic dissection [9]. On the other hand, ambiguities across databases may occur in C. elegans papers; for example, HID-1 is the name of a protein in C. elegans and in mouse [10]. In this case, WormBase links to HID-1 were selectively deleted when the authors were referencing the mouse HID-1.

An unintentional advantage of this pipeline is that our manual QC step also resolves author typos and XML formatting errors that occur during the document processing step at DJS; these types of errors result in wrong links or no link, so need to be dealt with in order to have the proper links made. We discovered these types of errors in at least 17 papers. We deal with these errors by alerting DJS who then contacts the author to sort out how to fix the typo. Because we are the main portal for C. elegans research and we are trained to spot details concerning variations, genes, transgenes etc., spotting these errors and understanding what the correction should be is not generally difficult. In most cases, we can go ahead and make the correct link to the typo and leave the author to request journal services to correct the text after we are through with the linking, so there is usually no waiting time involved in these situations.

On average it takes a WormBase curator 40 minutes to perform manual QC on a paper. The time required depends on many factors, the most important of which is the number of entity links generated in each paper. For example in five recent C. elegans articles, an average of 79 different unique URLs are generated for each article, many of which are propagated throughout the paper resulting in an average of 424 links per article; time is spent on ensuring that the initial URL is correct and that it is appropriate for the context.

The proper balance between automation and manual efforts

A recent article by Atwood et al. [17] presented a nice overview of the different hyperlinking tools currently available. While fully automated hyperlinking tools can provide instantaneous links, be portable enough to use on any online html page and can form links to any source, they are at a disadvantage when it comes to ensuring accuracy. By using a manual QC step we can selectively unlink ambiguous terms, ensuring that the reader is taken to the correct webpage. One suggested solution to resolve these ambiguities is to rely on user feedback and employ the reader to correct links, which is in use by Utopia Documents [14] and Reflect[1] (relayed to us by one of anonymous reviewers). While this may prove an optimal solution for these fully automated tools, as we are starting from the point of the actual database that is being linked to, we might not benefit as much as these other projects. However, we are open to modifications of our pipeline that would increase our efficiency. As the most time consuming steps of this pipeline are the manual QC and XML editing we are actively developing tools to cut down on these steps. For example, we have created an interface that allows the curator to more easily view the marked up html as well as a list of the entities and links.

Benefits to the community

We established a manual QC step to resolve ambiguous or erroneous links that occur with automated linking. Automated tools cannot distinguish one entity from another entity with the same name, even if the entities are, for example, genes in entirely different species. Because of the well-established nomenclature in the C. elegans field, ambiguity is low for most of the entity classes. As we begin to link terms in classes that use a more familiar term such as the anatomy class, we have noted a rise in the number of links to resolve.

We use our QC step to enrich our database by identifying entities missing from our database. We also can take advantage of the QC step to link synonyms of terms to the appropriate community-approved name. These synonyms could represent jargon terms or author assigned names that did not follow community-based nomenclature rules when first adopted. For example, a researcher may have cloned a gene and assigned a name to it without first finding out if the gene name conflicts with a pre-existing class of genes soon to be published. In such cases, a link between the gene name used by the author can be made to the correct sequence page at the database, offsetting any confusion should the gene name get changed after the article is published.

By far the most important duty of this step is to resolve ambiguous links; however we have taken advantage of this step to feed important information back to the authors and journal as well as to enhance our own database. For example, we use the author first pass form to capture new entities that don't exist in our database. Combined with the manual QC step, we are able incorporate data in our database before the paper is published. In addition, our manual QC step has identified entities that had been discovered years ago but had not been entered into the database yet, which occurred for two papers.

Finally, our manual QC step has proved beneficial to the authors and journal as we have been able to catch typos and XML formatting errors that were missed by the authors and copy-editors.

Resolving ambiguities

The hardest hurdle in all of the hyperlinking efforts to date is resolving which species an ambiguous entity belongs to (see [18] for example). One automated method, Linneaus, has been developed to tackle the problem of identifying all gene names belonging to all species in an article without any prior knowledge of which species are discussed [19, 20]. However, our problem differs from others in that we have an advantage of knowing the different species discussed in an article a priori from the journal publisher and the authors. Hence the problem of species identification itself is not a major challenge we face. In addition, as noted, authors usually specify the species name near the entity should there be any ambiguity [20]. For example, authors may use abbreviations (Sc for S. cerevisiae, Sp for Schizosaccharomyces pombe) before the entity name for disambiguation, which could be identified before linking is started. Finally, since we expand our pipeline one MOD at a time and work closely with MOD curators, identifying unique styles and conventions of writing scientific articles for each species could be captured and used for automatic disambiguation. While proximity of the species name to the entity does help in ambiguity resolution in most cases, such a heuristic approach may not work for all cases because of complexities in natural language texts. Ultimately, the manual QC step still remains the best way to identify and correct any errors arising out of automatic methods.

We are planning on linking articles from more model organisms and are looking for other databases and journals to actively participate in this project.