BLASTGrabber: a bioinformatic tool for visualization, analysis and sequence selection of massive BLAST data
© Neumann et al.; licensee BioMed Central Ltd. 2014
Received: 7 January 2014
Accepted: 31 March 2014
Published: 5 May 2014
Advances in sequencing efficiency have vastly increased the sizes of biological sequence databases, including many thousands of genome-sequenced species. The BLAST algorithm remains the main search engine for retrieving sequence information, and must consequently handle data on an unprecedented scale. This has been possible due to high-performance computers and parallel processing. However, the raw BLAST output from contemporary searches involving thousands of queries becomes ill-suited for direct human processing. Few programs attempt to directly visualize and interpret BLAST output; those that do often provide a mere basic structuring of BLAST data.
Here we present a bioinformatics application named BLASTGrabber suitable for high-throughput sequencing analysis. BLASTGrabber, being implemented as a Java application, is OS-independent and includes a user friendly graphical user interface. Text or XML-formatted BLAST output files can be directly imported, displayed and categorized based on BLAST statistics. Query names and FASTA headers can be analysed by text-mining. In addition to visualizing sequence alignments, BLAST data can be ordered as an interactive taxonomy tree. All modes of analysis support selection, export and storage of data. A Java interface-based plugin structure facilitates the addition of customized third party functionality.
The BLASTGrabber application introduces new ways of visualizing and analysing massive BLAST output data by integrating taxonomy identification, text mining capabilities and generic multi-dimensional rendering of BLAST hits. The program aims at a non-expert audience in terms of computer skills; the combination of new functionalities makes the program flexible and useful for a broad range of operations.
KeywordsAnalysis BLAST High-throughput Taxonomy Text-mining Visualization
Sequence similarity searches have become an integral part of contemporary life sciences . More than two decades have now passed since the Basic Local Alignment Search Tool (BLAST) was introduced to the bioinformatic community [2, 3], constituting a breakthrough for rapid similarity search tools . Despite the staggering changes that have taken place in biology, sequencing and computing technology, BLAST remains the most common used algorithm for sequence similarity searches [5, 6]. This is reflected by the exceptionally high numbers of citations for the two original BLAST papers (48632 citations , 49238 citations ; Google scholar - January 2014). The continued popularity is due to the intuitive appeal of the algorithm , its speed and efficiency [7, 8], and being supported by a complete, rigorous statistical theory .
BLAST has been used for most purposes involving biological sequence searches and alignments, some examples being EST annotation [4, 9–11], contig assembly [12, 13], genomic fragment reconstruction , ORF validation , prediction of protein function and origin [4, 6], distant homolog  and putative ortholog detection [15, 16], phylogenetic analysis [8, 17, 18] and metagenomics .
BLAST services can be accessed numerous places on the web and are often free of charge; one of most popular is the BLAST implementation hosted by the National Center for Biotechnology Information (NCBI) . It receives hundreds of thousands requests a day [21, 22], and presents BLAST results as textual reports with graphical representations of the calculated alignments.
Such web-based BLAST implementations are convenient to use for the analysis of a small number of sequences. However, both limitations of computational resources and the way results are presented render this approach unfeasible for large BLAST searches that might involve thousands (or even hundreds of thousands) of unique query sequences. In recent years, due to new sequencing technology, high-throughput searches of this magnitude have become a standard situation in many fields of research, such as EST annotation , genomics , metagenomics  and phylogenomics . In addition, projects without high-throughput sequencing data also exhibit a trend towards increasing query numbers. Examples might be all-against-all comparisons useful in EST annotation , proteome-against-proteome searches in order to identify orthologs , or simply pooling diverse queries together so as to minimize the number of job submissions to an external computing resource.
In order to perform the actual high-throughput BLAST search, it is necessary to use a stand-alone implementation of the BLAST algorithm rather than relying on web-based public installations such as the NCBI resource . Many research institutions have solved this by establishing high-performance computing (HPC) clusters hosting BLAST-related databases and pipelines. Since BLAST scales well when parallelized, such HPC installations can handle large high-throughput sequence input in a reasonable amount of time. The results of high-throughput BLAST runs could still present the user with gigabyte-sized text files, the data volume alone representing a challenge for inexperienced users. For specific research fields, massive BLAST output can be analysed by specialized, user-friendly programs that run on ordinary desktop computers – such as the MEGAN  program for metagenomics. For many other research fields however, scientists find suitable tools missing . Surprisingly, this includes generic BLAST output interpreter programs capable of visualizing, analysing and selecting massive BLAST output data.
This reflects a more general problem in the field of biological data visualization and analysis - contemporary biological data generation has outpaced the traditional data processing approaches [20, 24]. Some of the important features recently suggested to alleviate the perceived program shortcomings include (amongst others) improved usability, new visual analytical modes, and multi-scale standardized data representations . Especially intriguing is the challenge of visualizing massive amounts of data in a suitable way. New, innovative visual representation techniques are needed that provide both an overview over, and facilitates detailed navigation into, the mass of information in these data sets . Hence, improved visualization technologies are clearly amongst the key aspects of knowledge discovery and data mining ; some of which have begun to find their way into mainstream science (for instance, [26–28]). Suitable analysis tools will be in demand as sequencing technology become even more efficient.
We here present a program designed for the effective exploration of BLAST output generated from large scale database searches. It is aimed at an audience of computer non-experts not familiar with programming languages, database retrieval or command-line usage. The program facilitates visualization, analysis and selection of data. Importantly, the application provides new functionalities including taxonomic ordering of data, text search options, multi-dimensional display and a range of possibilities for filtering and downloading of data from public sequence databases. The program, introduction video and additional utilities are freely available for download at http://www.bioportal.no.
The BLASTGrabber application
The BLASTGrabber desktop application is a GUI-based Java program that will run on Windows-, MacOS- or Linux- based computers. Apart from the Java Runtime Environment (version 1.6 or higher), no other installations are required to run the program. BLASTGrabber will work on low-spec systems including netbooks. For high-throughput data however, 4 GB of memory and a processor comparable to 1.4GHz Core i5 are recommended. BLASTGrabber is installed by extracting the installation zip file containing the BLASTGrabber JAR file and additional files (such as example data sets and NCBI taxonomy file). In order to maximize memory usage, the program can be started from the command line, explicitly specifying the java heap space parameters. Alternatively, double-clicking an included shell file can start BLASTGrabber with 2 GB of maximum heap spaces.
BLASTGrabber uses a custom input file format (*.bgr file format) to represent BLAST output. This file is produced by an import wizard included in the BLASTGrabber application (for details, see User guide in Additional file 1). The wizard will recognize BLAST XML output format, in addition to standard text-formatted BLAST output from both the current BLAST + version and the older BLAST 2.2.* versions. As taxonomy information is not included in the standard BLAST output, taxonomy identifiers will not be included when importing BLAST files.
The *.bgr file is formatted as a text file, and lists the included queries and sequences (names or headers together with internal identifiers). Multiple hit sequences with identical FASTA headers are represented only once. In addition, the file contains BLAST statistics such as e-values and bit scores. NCBI TAXIDs can be appended to the *.bgr file after importing by using the included taxonomy wizard. These allow a downstream taxonomical representation in the BLASTGrabber program.
Upon loading a *.bgr file, the query, sequence and high scoring pair (HSP) attribute objects are instantiated from the *.bgr file and kept in the computer memory until the program is terminated (or another *.bgr file is loaded). Since all the BLAST data is directly available in computer memory, most BLASTGrabber operations complete in a few seconds at the most. Therefore installed memory limits the maximum BLAST data size usable by BLASTGrabber. For BLAST output exceeding these memory limitations, it is possible to break apart the offending BLAST file into multiple smaller parts. These can be imported and opened individually, and possibly merged together again at a later point in the analysis. The effectual maximum BLAST output size can be doubled using this technique.
The BLASTGrabber installation folder includes the NCBI taxonomy structure compressed into a custom binary format. In order to keep the start-up time and memory overhead as low as possible, only scientific taxonomy names are included. Loading the taxonomy happens automatically at BLASTGrabber start-up, but can be cancelled by the user in order to preserve memory or improve BLASTGrabber start-up time. The BLASTGrabber taxonomy file can be updated by downloading the NCBI taxonomy files and using the included update functionality.
BLASTGrabber includes text search options; these are implemented using the Java regular expression (regex) API. Graphical representations of hit alignments are based on HSP attributes describing the start and end of the BLAST-generated alignments. No mismatches or gaps are indicated since this information is not included in the *.bgr file. However, sequences and alignments for selected hits can be downloaded from NCBI or extracted from the original BLAST output and query files.
BLASTGrabber architecture supports third party plugin development not depending on BLASTGrabber source code. Communication to and from the plugin is defined by two specific interfaces; JAR folders in the “plugins” subfolder featuring these required interfaces are loaded and activated at BLASTGrabber start-up (API description and plugin example are provided at the BLASTGrabber download site).
Two queries representing the Fibronectin-3 (Pfam id: PF00041) and I-set (Pfam id: PF07679) domains on the Drosophila melanogaster “Down syndrome cell adhesion molecule” protein (uniprot id: A1Z6X1_DROME) were used in a BLAST search against the Drosophila uniprot reference proteome. These two motifs are found multiple times next to each other in various Drosophila multi-domain proteins and are well suited to illustrated sequence alignment visualization in BLASTGrabber.
The BLASTGrabber visualization and analysis of the example datasets was compared to the JAMBLAST  and BlastViewer (Korilog SARL, Questembert, France) programs. As JAMBLAST utilizes a customary csv format (*.bls) produced by the accompanying NOBLAST  BLAST implementation, identical searches were performed producing the required BLAST output formats. In addition to the NOBLAST *.bls format required by JAMBLAST, BLAST XML output is required for BlastViewer.
In order demonstrate taxonomy display, a BLAST search (using the BLOSUM45 matrix; expect threshold at e-value = 1) was performed against the NCBI “nr” database using the two “Fibronectin-3” and “I-set” queries mentioned above.
Results and discussion
User-friendly visualization and analysis
Despite the continuing high popularity of the BLAST algorithm and ever-increasing data loads, there is a perceived lack of high-throughput-capable, user-friendly generic BLAST output visualisation tools. BLASTGrabber is designed to be usable by computer non-experts and implements a truly multi-query mode-of-analysis which is backed by visual displays. The included taxonomy wizard adds taxonomy identifiers to the BLAST data, allowing the downstream integration of sequence descriptions, origins and hit quality assessment. Also, the BLASTGrabber architecture facilitates the development of third-party plugins, opening new possibilities for BLAST-related analysis algorithms. Some aspects of BLASTGrabber overlap with the JAMBLAST and BlastViewer programs functionalities. Just as BLASTGrabber, both are free and user-friendly Java programs that visualize BLAST files. While supporting many additional analysis features when compared to existing similar programs, we find BLASTGrabber to perform as well or better in terms of data input capacity and speed, all the time retaining minimal installation requirements and an intuitive and user-friendly interface.
BLASTGrabber uses a matrix viewer of hits and queries
This hit-focused (in contrast to query-focused) display mode allows the inspection of hundreds of thousands of hits at a glance, distributed across a few table cells only. Cell contents (i.e. the queries and hits) are easily selected, either by manually clicking on cells of interest, or else defining cut-off values for cell statistics (for instance selecting all hit with e-values below e-5). Selected cells are “grabbed”, i.e. copied to the BLASTGrabber clipboards. Clipboard contents are persistable as *.bgr files, allowing iterative purifying selection by *.bgr file opening, subset selection and persistence.In contrast to BLASTGrabber, the BlastViewer program displays hits and queries as a hierarchical structure. When using multiple-query BLAST output each query must be selected before the associated hits are displayed (Figure 2B). The hit list is sortable by any HSP attribute by simply clicking a column header; top-scoring hits can be exported to a text editor in csv format. Lexicographical sorting (for instance based on e-values, bit scores and identity percentages in this priority) can be accomplished by clicking column headers in the reverse order of the desired priority. BlastViewer visualizes all the BLAST attributes for each of the hits, but because queries have to be selected one at the time without any filtering options, it becomes virtually impossible to use for analysis of thousands, or even hundreds of thousand, queries.JAMBLAST also employs a hierarchical structure but has a slightly different visualization mode than BlastViewer, displaying all queries and top scoring hits in the same table (Figure 2C). The program supports lexicographical ordering by the explicit selection of the HSP attributes sort order. The best hits for multiple queries can be selected in one go with subsequent export to a text editor program. Without clustering multiple hits into fewer categories (such as dividing hits into significant and insignificant e-value categories), it is virtually impossible to extract relevant information from high-throughput data.
Most BLAST output interpreter programs provide a graphical representation of the BLAST-generated alignment, i.e. the sequence overlap between query and hit. In addition to displaying hits as they are aligned against a selected query, BLASTGrabber can also do the opposite; i.e. display multiple queries overlapping one selected hit. Thus, for instance the position of multiple genes (queries) along one chromosome (one hit) can be displayed.
Text searches with the description viewer
In addition to performing searches in the complete headers or query names, these can be split into individual words. This allows determination of the relative frequencies of keywords of interest, such as enzyme or species names. Word splitting is regex-based and fully configurable.
Taxonomical ordering of data in the matrix viewer
Splitting and merging files
Contemporary high-throughput projects may generate millions of sequences used as queries in BLAST similarity searches (for one example, see ). Depending on the installed computer memory, BLASTGrabber can handle BLAST output containing a few hundred thousand queries. If the relevant data set is too substantial to be imported into BLASTGrabber, the BLAST file can be split into smaller parts; each single part can then be imported individually into BLASTGrabber. Subsequently, several *.bgr files can be merged to join previously disjoint data sets.
Maximum data capacities of the BLAST interpreter programs*
XML file (BLAST XML format, size ~2.1 GB )
bls file (NOBLAST csv format, size ~290 MB)
bgr file (custom text format, size ~ 280 MB)
Performance examples of typical BLASTGrabber operations*
~90.000 queries (file size 640 MB )
Import (creation of *.bgr file)
Resulting *.bgr file
~45.000 queries** (file size 140 MB).
Opening the *.bgr file
Opening matrix viewer, 10 rows × 10 cols (100 cells)
Opening matrix viewer, all queries × 10 cols (450.000 cells)
Selecting a significant subset (e-value < e-5)
Saving the selected significant subset
Along with the rest of bioinformatics, performing BLAST searches has changed substantially over the past two decades. The default textual BLAST output is well adapted to single-query BLAST runs, but next-generation sequencing technologies and growing database sizes demand advanced visualization and analysis capabilities. At the same time, an increasing number of biologists without expert knowledge of programming languages or database retrieval techniques need to work with BLAST and BLAST-related tools.
We believe that the BLASTGrabber program can serve as such a tool. It enables the handling of large amounts of data on desktop computers, while at the same time delivering new and advanced visualization styles. In addition, the flexible text-mining options provide easy selection of BLAST hits or queries. Finally, the integrated taxonomy display can add an extra dimension to traditional sequence similarity searches.
Availability and requirements
Project name: BLASTGrabber.
Project home page: http://www.bioportal.no.
This web address also hosts the BLASTGrabber user guide and a tutorial video demonstrating the use of BLASTGrabber. The BLASTGrabber zip file contains both sample files and the user guide.
Operating system: Platform independent.
Programming language: Java.
Other requirements: Java Runtime Environment (version 1.6 or higher).
License: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
Any restrictions to use by non-academics: None.
Basic Local Alignment Search Tool
National Center for Biotechnology Information
University of Oslo.
We thank student and researchers at the Microbial Evolution Research Group (MERG) and Section for Genetics and Evolutionary Biology for testing of the BLASTGrabber program.
This work has been supported by a VISTA/Statoil fellowship to Thomas H. A. Haverkamp, PhD fellowships to Surendra Kumar and Ralf Neumann and starting grant to Kamran Shalchian-Tabrizi from the University of Oslo.
- Nilsson R, Larsson K, Ursing BM: Galaxie—CGI scripts for sequence identification through automated phylogenetic analysis. Bioinformatics. 2004, 20: 1447-1452. 10.1093/bioinformatics/bth119.View ArticlePubMedGoogle Scholar
- Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ: Basic local alignment search tool. J Mol Biol. 1990, 215: 403-410. 10.1016/S0022-2836(05)80360-2.View ArticlePubMedGoogle Scholar
- Altschul SF, Madden TL, Schäffer AA, Zhang J, Zhang Z, Miller W, Lipman DJ: Gapped BLAST and PSI-BLAST: a new generation of protein database search programs. Nucleic Acids Res. 1997, 25: 3389-3402. 10.1093/nar/25.17.3389.View ArticlePubMed CentralPubMedGoogle Scholar
- Kanehisa M, Bork P: Bioinformatics in the post-sequence era. Nat Genet. 2003, 33: 305-310. 10.1038/ng1109.View ArticlePubMedGoogle Scholar
- Koonin E: Sequence-Evolution-Function: Computational Approaches in Comparative Genomics. 2003, Dordrecht: Kluwer Academic PublishersView ArticleGoogle Scholar
- Dong Q, Brendel V: Computational Identification of Related Proteins. The Proteomics Protocols Handbook. Edited by: Walker JM. 2005, Totowa: Humana Press, 555-570.Google Scholar
- She R, Shih-Chieh Chu J, Uyar B, Wang J, Wang K, Chen N: genBlastG: using BLAST searches to build homologous gene models. Bioinformatics. 2011, 27: 2141-2143. 10.1093/bioinformatics/btr342.View ArticlePubMedGoogle Scholar
- Dereeper A, Audic S, Claverie J-M, Blanc G: BLAST-EXPLORER helps you building datasets for phylogenetic analysis. BMC Evol Biol. 2010, 10: 1-6. 10.1186/1471-2148-10-1.View ArticleGoogle Scholar
- Xing L, Brendel V: Multi-query sequence BLAST output examination with MuSeqBox. Bioinformatics. 2001, 17: 744-745. 10.1093/bioinformatics/17.8.744.View ArticlePubMedGoogle Scholar
- Pirooznia M, Perkins EJ, Deng Y: Batch Blast Extractor: an automated blastx parser application. BMC Genomics. 2008, 9: 1-5. 10.1186/1471-2164-9-1.View ArticleGoogle Scholar
- Zhao S, Shao C, Goropashnaya AV, Stewart NC, Xu Y, Tøien Ø, Barnes BM, Fedorov VB, Yan J: Genomic analysis of expressed sequence tags in American black bear Ursus americanus. BMC Genomics. 2010, 11: 201-10.1186/1471-2164-11-201.View ArticlePubMed CentralPubMedGoogle Scholar
- Koltes J, Hu Z, Fritz E: BEAP: The BLAST Extension and Alignment Program-a tool for contig construction and analysis of preliminary genome sequence. BMC Res Notes. 2009, 2: 11-10.1186/1756-0500-2-11.View ArticlePubMed CentralPubMedGoogle Scholar
- Paquola ACM, Machado AA, Reis EM, da Silva AM, Verjovski-Almeida S: Zerg: a very fast BLAST parser library. Bioinformatics. 2003, 19: 1035-1036. 10.1093/bioinformatics/btg122.View ArticlePubMedGoogle Scholar
- Suyama M, Torrents D, Bork P: BLAST2GENE: a comprehensive conversion of BLAST output into independent genes and gene fragments. Bioinformatics. 2004, 20: 1968-1970. 10.1093/bioinformatics/bth172.View ArticlePubMedGoogle Scholar
- Wall DP, Fraser HB, Hirsh AE: Detecting putative orthologs. Bioinformatics. 2003, 19: 1710-1711. 10.1093/bioinformatics/btg213.View ArticlePubMedGoogle Scholar
- Zhao S, Burki F, Keeling P: Collodictyon – an ancient lineage in the tree of eukaryotes. Mol Biol Evol. 2012, 29: 1557-1568. 10.1093/molbev/mss001.View ArticlePubMed CentralPubMedGoogle Scholar
- Koski LB, Golding GB: The closest BLAST hit is often not the nearest neighbor. J Mol Evol. 2001, 52: 540-542. 10.1007/s002390010184.View ArticlePubMedGoogle Scholar
- Shalchian-Tabrizi K, Minge MA, Espelund M, Orr R, Ruden T, Jakobsen KS, Cavalier-Smith T: Multigene phylogeny of choanozoa and the origin of animals. PLoS ONE. 2008, 3: 1-7.View ArticleGoogle Scholar
- Huson DH, Auch AF, Qi J, Schuster SC: MEGAN analysis of metagenomic data. Genome Res. 2007, 17: 377-386. 10.1101/gr.5969107.View ArticlePubMed CentralPubMedGoogle Scholar
- Procter JB, Thompson J, Letunic I, Creevey C, Jossinet F, Barton GJ: Visualization of multiple alignments, phylogenies and gene family evolution. Nat Publ Group. 2010, 7: S16-S25.Google Scholar
- Mount DW: Bioinformatics: Sequence and Genome Analysis. 2004, New York: Cold Spring Harbor Laboratory PressGoogle Scholar
- Weintraub B: Building BLAST for Coprocessor Accelerators Using Macah. PhD thesis. 2008, University of Washington, Computer Science And EngineeringGoogle Scholar
- Decker JE, Pires JC, Conant GC, McKay SD, Heaton MP, Chen K, Cooper A, Vilkki J, Seabury CM, Caetano AR: Resolving the evolution of extant and extinct ruminants with high-throughput phylogenomics. Proc Natl Acad Sci. 2009, 106: 18644-18649. 10.1073/pnas.0904691106.View ArticlePubMed CentralPubMedGoogle Scholar
- O’Donoghue SI, Gavin A-C, Gehlenborg N, Goodsell DS, Hériché J-K, Nielsen CB, North C, Olson AJ, Procter JB, Shattuck DW, Walter T, Wong B: Visualizing biological data — now and in the future. Nat Publ Group. 2010, 7: S2-S4.Google Scholar
- Fayyad U, Piatetsky-Shapiro G, Smyth P: From data mining to knowledge discovery in databases. AI Mag. 1996, 17: 37-54.Google Scholar
- Darzentas N: Circoletto: visualizing sequence similarity with circos. Bioinformatics. 2010, 26: 2620-2621. 10.1093/bioinformatics/btq484.View ArticlePubMedGoogle Scholar
- Otto A, Bernhardt J, Meyer H, Schaffer M, Herbst F-A, Siebourg J, Mäder U, Lalk M, Hecker M, Becher D: Systems-wide temporal proteomic profiling in glucose-starved Bacillus subtilis. Nat Commun. 2010, 1: 1-9.View ArticleGoogle Scholar
- Krzywinski M, Birol I, Jones SJ, Marra M: Hive plots--rational approach to visualizing networks. Brief Bioinform. 2011, 13: 1-18.Google Scholar
- Lagnel J, Tsigenopoulos CS, Iliopoulos I: NOBLAST and JAMBLAST: New Options for BLAST and a Java Application Manager for BLAST results. Bioinformatics. 2009, 25: 824-826. 10.1093/bioinformatics/btp067.View ArticlePubMedGoogle Scholar
- Nilsson RH, Kristiansson E, Ryberg M, Larsson KH: Approaching the taxonomic affiliation of unidentified sequences in public databases–an example from the mycorrhizal fungi. BMC Bioinforma. 2005, 6: 178-10.1186/1471-2105-6-178.View ArticleGoogle Scholar
- Pozhitkov A: Molecular taxonomy. Bioinformatics and practical evaluation. PhD thesis. 2003, Universitet Köln, Mathematisch-Naturwissenschaftliche FakultätGoogle Scholar
- Coin L, Bateman A: Enhanced protein domain discovery using taxonomy. BMC Bioinforma. 2004, 5: 56-10.1186/1471-2105-5-56.View ArticleGoogle Scholar
- Yang JY, Chen X: Improving taxonomy-based protein fold recognition by using global and local features. Proteins Struct Funct Bioinformatics. 2011, 79: 2053-2064. 10.1002/prot.23025.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.