Volume 6 Supplement 4
ParPEST: a pipeline for EST data analysis based on parallel computing
© D'Agostino et al; licensee BioMed Central Ltd 2005
Published: 1 December 2005
Expressed Sequence Tags (ESTs) are short and error-prone DNA sequences generated from the 5' and 3' ends of randomly selected cDNA clones. They provide an important resource for comparative and functional genomic studies and, moreover, represent a reliable information for the annotation of genomic sequences. Because of the advances in biotechnologies, ESTs are daily determined in the form of large datasets. Therefore, suitable and efficient bioinformatic approaches are necessary to organize data related information content for further investigations.
We implemented ParPEST (Par allel P rocessing of EST s), a pipeline based on parallel computing for EST analysis. The results are organized in a suitable data warehouse to provide a starting point to mine expressed sequence datasets. The collected information is useful for investigations on data quality and on data information content, enriched also by a preliminary functional annotation.
The pipeline presented here has been developed to perform an exhaustive and reliable analysis on EST data and to provide a curated set of information based on a relational database. Moreover, it is designed to reduce execution time of the specific steps required for a complete analysis using distributed processes and parallelized software. It is conceived to run on low requiring hardware components, to fulfill increasing demand, typical of the data used, and scalability at affordable costs.
The role of bioinformatics to support the Life Sciences has become fundamental for the collection, the management and the interpretation of large amount of biological data. The data are in most cases derived from experimental methodologies with large scale approaches, the so-called "omics" projects. International projects aimed to sequence the whole genomes of model organisms are often paralleled by initiatives for the expressed data sequencing to support gene identification and functional characterizations. Moreover, because of advances in biotechnologies, ESTs are daily determined in the form of large datasets from many different laboratories. Therefore, the analyses of expressed sequence data involve the necessity of suitable and efficient methodologies to provide high quality information for further investigations. Furthermore, suitable models for the organization of information related to EST data collections are fundamental to provide a preliminary environment for analyses of structural features of the data, as well as of expression maps and of functional relationships useful for the interpretation of mechanisms and of rules of gene expression processes.
There are many software available for EST processing, with the purpose to clean the datasets from contaminations [1–4] and to cluster sequences sharing identities to assemble contigs [5–10]. Sequences cleaned from contaminations are usually submitted to the dbEST database as they represent a fundamental source of information for the scientific community [11–13]. The results of the clustering step are useful to analyse sequence redundancy and variants as they could represent products of the same gene or of gene families. Moreover, ESTs or contigs obtained from the clustering step are usually analysed by comparisons with biological databanks to provide preliminary functional annotations . On the other hand, few efforts are known where all the sets of consecutive steps for EST processing, clustering and annotation are integrated into a single procedure [15–17].
Expressed sequence curated databanks are worldwide available. They consist of collections built starting from dbEST, using selected computational tools to solve the complex series of consecutive analyses. Some of the well known efforts are the Unigene database [18, 19], the TIGR gene indices  and the STACK project [21, 22].
Our contribution to this research is a pipeline, named ParPEST (Par allel P rocessing of EST s), for the pre-processing, clustering, assembling and preliminary annotation of ESTs, based on parallel computing and on automatic information storage. Useful information resulting from each single step of the pipeline is integrated into a relational database and can be analysed by Structured Query Language (SQL) calls for a "ad hoc" data-mining. We also provide a web interface with suitable pre-defined queries to the database for interactive browsing of the results that is supported by graphical views.
The pipeline has been implemented using public software integrated by in-house developed Perl scripts, on a 'Beowulf class' cluster, with Linux (Red Hat Fedora Core 2) as default operating system and the OSCAR 4.0 distribution  that provides the tools and the software packages for cluster management and parallel job executions.
The main process of the pipeline is designed to serialize and to control the parallel execution of the different steps required for the analysis and to parse into reports the collected results.
Input datasets are parsed by a specific routine so that information from the GenBank format or included in the FASTA format could be upload into a MySQL database.
Sequence data are pre-processed in two steps, to clean the data and to avoid mis-clustering and/or mis-assembling. The first step requires RepeatMasker  and the NCBI's VECTOR database  for checking vector contaminations. In the second step, RepeatMasker and RepBase  are used for filtering and masking low complexity sub-sequences and interspersed repeats. To accomplish sequence pre-processing a specific utility has been designed to distribute the tasks across the computing nodes. Job assignments are managed by a PBS batch system . Job control at each step and output files integration is managed by the main process.
PaCE  is the software we selected for the clustering step. For a parallel execution it requires an MPI implementation and a job scheduler server. Once the whole pre-processed sequences are clustered, they are assembled into contigs using CAP3 . To exploit the efficiency of CAP3 and to avoid the overhead time consuming of PBS, the main process we implemented has been designed to bundle groups of commands to be executed sequentially by each processor.
The functional annotation is performed using the MPI-Blast package . Raw EST data and assembled contigs are compared using BLASTx versus UNIPROT database . The blast search is performed setting an E-value less equal than 1. In case of successful matches, the five best hits are reported. When the subject accession number is reported in the Gene Ontology (GO) database [29, 30] the corresponding classification is included to further describe the putative functionalities. Moreover, links to the KEGG database  are provided via the ENZYME  identifier in the resulting report, for investigations on metabolic pathways.
All the results obtained from the single steps of the pipeline are recorded in a relational database and are managed through SQL calls implemented in a suitable PHP-based web interface to allow interactive browsing of all the structural features of each EST, their organization in the assembled contigs, the BLAST-derived annotations as well as the GO classifications.
Results and discussion
The pipeline performs a parallel analysis on large amount of EST data. Because of distributed computing there is no execution limit for the processes, that are allocated according to available resources. The free release of PaCE,  that we experienced to be limited at 30.000 sequences, has been updated with the latest version provided by the authors who successfully tested the software with more than 200.000 sequences (personnel communication). Therefore, the only limiting factor for the complete execution of the pipeline is the memory space required for the database storage.
Execution times for different node and data-set configurations. The execution times (in seconds) are collected for each step of the pipeline: 1) Blast on ESTs: functional annotation of raw EST sequences; 2) Pre-processing: vector contaminations cleaning and low complexity and interspersed repeat sequences masking; 3) Clustering; 4) Assembling; 5) Blast on Contigs: functional annotation of consensus sequences. Tot: is the global execution time of the pipeline.
Blast on ESTs
Blast on Contigs
We implemented the software to make it independent of the resource manager server. Therefore, though we based the system on a PBS resource manager, it can be easily ported to other environment such as Globus Toolkit  or SUN Grid Engine (SGE) . Therefore the current pipeline could be also implemented on to the latest GRID computing environment.
Another table is used to describe vector contaminations according to the report the main process of the pipeline automatically produces during the pre-processing step (Figure 1). Therefore the database will include information about the ESTs still including vector or linker contaminating sequences. A similar approach is used to report masked regions representing low complexity subsequences or repeats as identified byRepeatMasker, using RepBase as the filtering database.
Clusters obtained from PaCE resulting in single EST sequence or in contigs, are collected in the database too. A specific routine included in the main process of the pipeline performs a deep analyses on the clustered sequence to derive information on how many ESTs belong to a single contig, and how many contigs are produced once the sequences are clustered.
CAP3 assemblies sequences building a multiple alignment and deriving a consensus to obtain a contig. To use only high-quality reads during assembly, CAP3 removes automatically 5' and 3' low-quality regions (clipping step). Therefore, to keep information about the whole assembly process, both the complete alignment and the EST trimmed regions are recorded into the database.
The table designed to organize the BLAST report from raw EST data as well as from contig sequence analyses, can include the five most similar subject sequences and their related information. Gene Ontology terms related to each BLAST hit are recorded into the GO table included in the database.
Users can further select data based on the preliminary functional annotation, specifying a biological function as well as a GO term or a GO accession (Figure 5b). Moreover, restrictions on results obtained from the whole analytical procedure can be applied to retrieve different sets of ESTs (Figure 5c). For example users can retrieve all ESTs containing or not vectors, presenting or not BLAST matches, classified as singletons or to be in a cluster.
We designed the presented pipeline to perform an exhaustive analysis on EST datasets. Moreover, we implemented ParPEST to reduce execution time of the different steps required for a complete analysis by means of distributed processing and of parallelized software. Though some efforts are reported in the literature where all the steps included in a EST comprehensive analyses are integrated in a pipelined approach [11–13], to our knowledge, no public available software is based on parallel computing for the whole data processing. The time efficiency is very important if we consider that EST data are in continuous upgrading.
The pipeline is conceived to run on low requiring hardware components, to fulfill increasing demand, typical of the data used, and scalability at affordable costs.
Our efforts has been focused to fulfill all the possible automatic analyses useful to highlight structural features of the data and to link the resulting data to biological processes with standardized annotation such as Gene Ontology and KEGG. This is fundamental to contribute to the comprehension of transcriptional and post-transcriptional mechanisms and to derive patterns of expression, to characterize properties and relationships and uncover still unknown biological functionalities.
Our goal was to set up an integrated computational platform, exploiting efficient computing, including a comprehensive informative system and ensuring flexible queries on varied fundamental aspects, also based on suitable graphical views of the results, to support exhaustive and faster investigations on challenging biological data collections.
The design of the platform is conceived to provide the pipeline and its results using a user friendly web interface. Upon request, users can upload GenBank or Fasta formatted files.
We offer free support for processing sequence collections to the academic community under specific agreements. We would welcome you to find contacts and to visit a demo version of the web interface at http://www.cab.unina.it/parpest/demo/.
This work is supported by the Agronanotech Project (Ministry of Agriculture, Italy).
We thank Prof. Luigi Frusciante and Prof. Gerardo Toraldo for all their support to our work.
We thank Anantharaman Kalyanaraman for his suggestions and updates about PaCE and Enrico Raimondo for useful discussions.
- Chou HH, Holmes MH: DNA sequence quality trimming and vector removal. Bioinformatics 2001, 17: 1093–104. 10.1093/bioinformatics/17.12.1093View ArticlePubMedGoogle Scholar
- SeqClean a software for vector trimming[http://www.tigr.org/tdb/tgi/software/]
- PHRAP software[http://www.phrap.org/]
- RepeatMasker software[http://www.repeatmasker.org/]
- Zhang Z, Schwartz S, Wagner L, Miller W: A greedy algorithm for aligning DNA sequences. J Comput Biol 2000, 7: 203–214. 10.1089/10665270050081478View ArticlePubMedGoogle Scholar
- Kalyanaraman A, Aluru S, Kothari S, Brendel V: Efficient clustering of large EST data sets on parallel computers. Nucleic Acids Res 2003, 31: 2963–2974. 10.1093/nar/gkg379PubMed CentralView ArticlePubMedGoogle Scholar
- Burke J, Davison D, Hide W: d2_cluster: a validated method for clustering EST and full-length cDNAsequences. Genome Res 1999, 9: 1135–1142. 10.1101/gr.9.11.1135PubMed CentralView ArticlePubMedGoogle Scholar
- Malde K, Coward E, Jonassen I: A graph based algorithm for generating EST consensus sequences. Bioinformatics 2005, 21: 1371–1375. 10.1093/bioinformatics/bti184View ArticlePubMedGoogle Scholar
- Parkinson J, Guiliano DB, Blaxter M: Making sense of EST sequences by CLOBBing them. BMC Bioinformatics 2002, 3: 31. 10.1186/1471-2105-3-31PubMed CentralView ArticlePubMedGoogle Scholar
- Pertea G, et al.: TIGR Gene Indices clustering tools (TGICL): a software system for fast clustering of large EST datasets. Bioinformatics 2003, 19: 651–652. 10.1093/bioinformatics/btg034View ArticlePubMedGoogle Scholar
- Boguski MS, Lowe TM, Tolstoshev CM: dbEST-database for "expressed sequence tags". Nat Genet 1993, 4: 332–333. 10.1038/ng0893-332View ArticlePubMedGoogle Scholar
- EGTDC: EST analysis[http://envgen.nox.ac.uk/est.html]
- Mao C, Cushman JC, May GD, Weller JW: ESTAP – an automated system for the analysis of EST data. Bioinformatics 2003, 19: 1720–1722. 10.1093/bioinformatics/btg205View ArticlePubMedGoogle Scholar
- Hotz-Wagenblatt A, Hankeln T, Ernst P, Glatting KH, Schmidt ER, Suhai S: ESTAnnotator: A tool for high throughput EST annotation. Nucleic Acids Res 2003, 31: 3716–3719. 10.1093/nar/gkg566PubMed CentralView ArticlePubMedGoogle Scholar
- Rudd S: openSputnik – a database to ESTablish comparative plant genomics using unsaturated sequence collections. Nucleic Acids Res 2005, 33: D622-D627. 10.1093/nar/gki040PubMed CentralView ArticlePubMedGoogle Scholar
- Kumar CG, LeDuc R, Gong G, Roinishivili L, Lewin HA, Liu L: ESTIMA, a tool for EST management in a multi-project environment. BMC Bioinformatics 2004, 5: 176. 10.1186/1471-2105-5-176PubMed CentralView ArticlePubMedGoogle Scholar
- Xu H, et al.: EST pipeline system: detailed and automated EST data processing and mining. Genomics Proteomics Bioinformatics 2003, 1: 236–242.PubMedGoogle Scholar
- Pontius JU, Wagner L, Schuler GD: UniGene: a unified view of the transcriptome. The NCBI Handbook 2003.Google Scholar
- Boguski MS, Schuler GD: ESTablishing a human transcripit map. Nat Genet 1995, 10: 369–371. 10.1038/ng0895-369View ArticlePubMedGoogle Scholar
- Lee Y, Tsai J, Sunkara S, Karamycheva S, Pertea G, Sultana R, Antonescu V, Chan A, Cheung F, Quackenbush J: The TIGR Gene Indices: clustering and assembling EST and known genes and integration with eukaryotic genomes. Nucleic Acids Res 2005, 33: D71-D74. 10.1093/nar/gki064PubMed CentralView ArticlePubMedGoogle Scholar
- Christoffels A, van Gelder A, Greyling G, Miller R, Hide T, Hide W: STACK : Sequence Tag Alignment and Consensus Knowledgebase. Nucleic Acids Res 2001, 29: 234–238. 10.1093/nar/29.1.234PubMed CentralView ArticlePubMedGoogle Scholar
- Miller RT, Christoffels AG, Gopalakrishnan C, Burke J, Ptitsyn AA, Broveak TR, Hide WA: A comprehensive approach to clustering of expressed human gene sequence: the sequence tag alignment and consensus knowledge base. Genome Res 1999, 9: 143–155. 10.1101/gr.9.11.1143View ArticleGoogle Scholar
- TORQUE PBS implementation[http://www.clusterresources.com/products/torque/]
- NCBI VECTOR Database[ftp://ftp.ncbi.nih.gov/blast/db/FASTA/vector.gz]
- Jurka J: Repbase Update: a database and an electronic journal of repetitive elements. Trends Genet 2000, 9: 418–420. 10.1016/S0168-9525(00)02093-XView ArticleGoogle Scholar
- Huang X, Madan A: CAP3: A DNA sequence assembly program. Genome Res 1999, 9: 868–877. 10.1101/gr.9.9.868PubMed CentralView ArticlePubMedGoogle Scholar
- MPIBlast software[http://mpiblast.lanl.gov/]
- Apweiler A, et al.: UniProt: the Universal Protein Knowledgebase. Nucleic Acids Res 2004, 32: D115-D119. 10.1093/nar/gkh131PubMed CentralView ArticlePubMedGoogle Scholar
- The Gene Ontology Consortium: Gene Ontology: tool for the unification of biology. Nat Genet 2000, 25: 25–29. 10.1038/75556PubMed CentralView ArticleGoogle Scholar
- The Gene Ontology Consortium: The Gene Ontology (GO) database and informatics resource. Nucleic Acids Res 32: D258-D261. 10.1093/nar/gkh036Google Scholar
- Kanehisa M, Goto S, Kawashima S, Okuno Y, Hattori M: The KEGG resources for deciphering the genome. Nucleic Acids Res 2004, 32: D277-D280. 10.1093/nar/gkh063PubMed CentralView ArticlePubMedGoogle Scholar
- Bairoch A: The ENZYME database in 2000. Nucleic Acids Res 28: 304–305. 10.1093/nar/28.1.304Google Scholar
- Globus toolkit[http://www-unix.globus.org/toolkit/]
- Sun Grid Engine[http://gridengine.sunsource.net/]
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.