- Methodology article
- Open Access
Booly: a new data integration platform
© Do et al; licensee BioMed Central Ltd. 2010
Received: 15 June 2010
Accepted: 13 October 2010
Published: 13 October 2010
Data integration is an escalating problem in bioinformatics. We have developed a web tool and warehousing system, Booly, that features a simple yet flexible data model coupled with the ability to perform powerful comparative analysis, including the use of Boolean logic to merge datasets together, and an integrated aliasing system to decipher differing names of the same gene or protein. Furthermore, Booly features a collaborative sharing system and a public repository so that users can retrieve new datasets while contributors can easily disseminate new content.
We illustrate the uses of Booly with several examples including: the versatile creation of homebrew datasets, the integration of heterogeneous data to identify genes useful for comparing avian and mammalian brain architecture, and generation of a list of Food and Drug Administration (FDA) approved drugs with possible alternative disease targets.
The Booly paradigm for data storage and analysis should facilitate integration between disparate biological and medical fields and result in novel discoveries that can then be validated experimentally. Booly can be accessed at http://booly.ucsd.edu.
The difficulty in achieving lasting solutions to integration of diverse biological data continues to be a central problem in bioinformatics [1, 2]. A number of technologies and systems have been developed that offer a variety of potential solutions to the data integration problem. These solutions differ by the architecture they adopt (data warehousing [BioMART, BioWarehouse], link integration [SRS, Entrez, TAMBIS, BioZon], semantic integration [BioMoby, Bio2RDF]) and by the common "touch-points" used to integrate data (e.g., data values, names, identities, schema properties, ontology terms, Uniform Resource Identifier [URI], keywords, loci, spatial-temporal points) [1, 3–11].
One major hurdle in current data integration efforts is the issue of naming and identity such that a variety of aliases exist for many of the same genes, proteins, and keywords. For example, Goble et al. states: "The failure to address identity will be the most likely obstacle that will stop mashups, or any other technology or strategy, becoming an effective integration mechanism." Another shortcoming of current data integration architectures is the high barrier of entry for contributions from both developers and the general research public. The data integration systems currently in existence either do not account for the general researcher contribution or are too difficult to utilize by non-specialists.
We have developed a data integration platform with an easy to use web interface that allows a broad variety of users to perform powerful comparative analysis between disparate datasets, including the use of Boolean operations (union, intersection, subtraction) and concatenated comparisons. An important element of this system is an integrated streamlined aliasing system to address the identifier problem by deciphering differing names of the same gene or protein. Furthermore, we have created a simple yet flexible data model so that the barrier of entry for both developers and researchers is minimal. For example, researchers can easily create new datasets by cutting and pasting their data from spreadsheet tables, while developers will have direct access through our Representational State Transfer (REST) style web services. Finally, we have created a collaborative sharing system and a repository within Booly so that users can retrieve new datasets while contributors can easily and securely share new content, allowing for fast creation and dissemination of entirely new databases with minimal effort.
We anticipate that the novel framework Booly provides for storing and integrating biological databases, with contributions from general researchers and large data centers alike coupled with high-throughput on demand alias translations, will spark a new approach in data integration efforts. These advantages over existing data integration approaches should attract growing contributions from developers and the research community that spur important new discoveries.
Privacy and Data Integrity
The Booly web application allows for users to create a secure personalized account for storage of datasets. In this manner, only the original owner of a data set will be able to view, modify, delete, and share their content. Once a data set is shared either publicly or to other individuals, permission is granted for the recipients to receive a copy of the data set, thereby preserving the original data set's integrity. The security of individual accounts is consistent with today's current web standards and will continually see improvements as the technology advances.
Extensions and Applications
In addition to the core functionalities we have previously described, Booly can be extended further by creation of new applications. For example, we created an application that allows researchers to generate new BLAST datasets. Another application allows the user to switch "touch points" (identifiers used to map one piece of data to another) , which makes it possible to perform concatenated series of complex Boolean comparisons. An example of the utility of this tool is to integrate known Drosophila melanogaster genetic interaction networks with human diseases and existing uses of FDA approved drugs to develop a new approach to identify new potential uses for drugs, sometimes referred to as drug repurposing (Additional file 1 [Suppl. Results], Additional file 2 [Figure S3], Additional file 2 [Figure S4]).
Additionally, we are in the process of creating an Application Programming Interface (API) utilizing RESTful web services (http://booly.ucsd.edu/api), which will allow developers an easy way to both import and retrieve data within Booly.
The growing volume of biological and medical information deposited within disparate databases has created an organization and data integration dilemma within the research community. Furthermore, new data not configuring to pre-existing specialized databases must await creation of new dedicated inclusive databases. We have created a novel tool, Booly, as a web application that solves key problems impeding current data integration efforts. An important feature of this system is a real time alias translation system, which we used to successfully integrate datasets with heterogeneous identifiers between Ensembl, gene symbols and gene ontology. Secondly, we addressed the issue of the entry barrier by creating an easy to use contribution model for both developers and researchers. Users are able to easily add datasets by copying and pasting their spreadsheet tables or by utilizing applications designed to create new Booly datasets. Lastly, we showed how Booly could be used as an intermediate step in data mining and data integration through our implementation of the switching and chaining technique to change "touch points".
There are a myriad of other enabling applications for Booly. For example, as personalized genomes become available to the general population, Booly is poised to offer individuals space to house their biological and medical information such that it can also be used to compare with publicly available content in a safe and secure fashion. Booly is also a resource for developers to add content without the obstacle of creating an online storage facility or the troublesome nature of alias resolution. Booly thus offers a fundamentally new paradigm for storing, sharing, and integrating current and future health and biological content.
Boolean modeling is a formal description of a broad array of biological phenomena, one notable example being gene regulation . To this extent, many biological processes can be modeled by using Boolean Networks. Booly offers an important functionality for system level studies as it greatly facilitates integration of diverse datasets from multiple experimental sources, providing the first step in gathering data into a Boolean model. Further development of algorithms that apply networking or clustering of touch points within the groupings created by Booly could similarly lead to novel systems based hypotheses.
Booly offers a Uniform Resource Locator (URL) based web API, allowing developers to easily integrate their applications and datasets into Booly. In this manner, developers will be able to create their tool or database and use Booly as a repository for the tool's output. For example, an external database may allow users to directly download all of the results from a search and place them directly into the user's Booly account. The output generated from these tools, once placed inside Booly, will inherit all its functionality, including the ability to easily share the data, to perform Boolean logic comparisons with other data sets, and to resolve aliases.
An obvious concern for comprehensive databases, and thus for Booly, is the issue of scalability. That is, how will Booly deal with the exponential growth of data deposited into its systems? For example, as personalized genomes become a reality, as is currently being implemented in the 1000 Genomes Initiative , a means for an individual to store and explore this information will be highly desirable. We have created Booly in such a way that as the data grows, additional machines can be introduced in parallel into the system for load balancing and data partitioning without adversely affecting the Booly's efficiency (speed) and reliability (uptime).
A large component of Booly is the user contribution model as similarly applied to such online applications as Wikipedia and more relevantly, WikiGene . However, a major concern is quality control of user-contributed data. Our plan to address this dilemma is to implement a community based review system for each dataset (similar to Amazon product ratings). In this manner, users will be able to search and add datasets based on "collective intelligence", a key element of Web 2.0 .
With most if not all data integration platforms, there is a concern that data can quickly grow out of date and require updating. The goal for Booly is to become a publically driven data repository reviewed and updated by its community. To aid the community in their efforts, we hope to implement a notification system such that when a new dataset is available, subscribers to the old dataset will be notified and allowed to upgrade or add the new dataset. In the meantime, we have created a forum message board so that contributors can disseminate update information to the community.
Finally, there is a growing movement in the life sciences to develop tools for semantic integration by way of the Resource Description Framework (RDF) model [11, 23]. Semantic integration approaches involve establishing complex relationships between objects, which can then be used to classify them or extract novel information regarding their behaviours. The goal of Booly is more modest, to establish identity between objects and to use this information to integrate data in which distinct names refer to the same objects. We felt our initial challenge was to help researchers and developers get their data quickly onto the web and to address the identity problem directly. However, to aid in interoperability with other data integration efforts that utilize RDF and other semantic integration approaches, we plan to provide export of data into a structured model such as RDF. It is our hope that the streamlined but efficient and user friendly comparative tools offered by Booly attract a broad base of users who are confronted by the simple but vexing problem of integrating data from a diverse set of spread sheets. Such users once adept at using Booly would presumably be primed to expand their sphere of comparison by trying out new tools such as those offered by semantic integration approaches.
Booly offers a new platform for the creation, storage, and integration of both personalized and public biological databases. As more applications are developed around the Booly platform, we anticipate these additions will further enhance the user experience. Booly presents a great opportunity to engage the research community in sharing data and adding combinatorial depth to potential queries. Such advances as offered by Booly should greatly aid researchers in formulating new questions that lead to novel discoveries in the laboratory.
Availability and Requirements
Design of Booly
Booly is comprised of the same components found in many current cloud computing web applications. These components can run on multiple servers and consist of a relational database that is load balanced and horizontally partitioned, custom scripts that access the database and perform Boolean operations, and a graphical user interface that can be accessed using a web browser. An overview of the Booly integration algorithm is shown in Figure 1. Further details of the Booly infrastructure and data integration components are described in Additional file 1 (Suppl. Methods).
Booly Integration Algorithm
Dataset Ordering. Boolean operations are performed based on the order of precedence as described in Figure 2a.
Alias Hash Key Conversion. When aliasing is requested, all identifiers from every dataset (D 1..n ) are converted to a hash key from an in-house Alias lookup database. The hash key is derived by utilizing the Secure Hashing Algorithm 2 (SHA2) 160-bit digest of a fingerprint such as a gene sequence, chemical formula, URI, etc... (Figure 3, Additional file 1 [Suppl. Methods]). The hash key is unique to the fingerprint (avoiding collisions as is the problem with, e.g. numerical identifiers) and can convert any arbitrary length message into 40 hexadecimal characters. This makes the hash key ideal as a non-semantic identifier.
Identifiers Grouped Based on Aliases. Hash Keys as well as the original identifiers are grouped based on exact matches. Groups are then consolidated based on the criteria that identifiers are one and the same when the same hash key exists amongst all identifiers in question.
Consolidated Groups Undergo Boolean Operation. The first pair of datasets (D 1 , D 2 ) based on Step 1 undergo the requested Boolean operation. The operation is performed iteratively until all matched aliases between the two datasets are exhausted.
Datasets Combined. The results of Step 4 are combined into a temporary dataset (D 1,2 ). D 1,2 is compared against D 3 and steps 4-5 are repeated until a final dataset D 1..n emerges.
We would like to thank Steve Wasserman, Russell Doolittle, Jotun Heine, Colleen Doherty, Steve Kay, Steve Briggs, Tom Watts, Teresa Markow, Phillip Bourne and members of the Bier Lab for helpful comments on the manuscript. We also thank Amir Gamliel for help with RTqPCR experiments and Agnieszka Brzozowska-Prechtl for assistance with dissection of chick brain regions. We thank Suchit Jhunjhunwala for insightful discussions on algorithmic and development aspects of Booly.
Funding: NIH RO1 AI070654 and NS29870
- Goble C, Stevens R: State of the nation in data integration for bioinformatics. J Biomed Inform 2008, 41(5):687–693. 10.1016/j.jbi.2008.01.008View ArticlePubMedGoogle Scholar
- Stein LD: Integrating biological databases. Nat Rev Genet 2003, 4(5):337–345. 10.1038/nrg1065View ArticlePubMedGoogle Scholar
- Haider S, Ballester B, Smedley D, Zhang J, Rice P, Kasprzyk A: BioMart Central Portal--unified access to biological data. Nucleic Acids Res 2009, (37 Web Server):W23–27. 10.1093/nar/gkp265Google Scholar
- Lee TJ, Pouliot Y, Wagner V, Gupta P, Stringer-Calvert DW, Tenenbaum JD, Karp PD: BioWarehouse: a bioinformatics database warehouse toolkit. BMC Bioinformatics 2006, 7: 170. 10.1186/1471-2105-7-170View ArticlePubMedPubMed CentralGoogle Scholar
- Birkland A, Yona G: BIOZON: a hub of heterogeneous biological data. Nucleic Acids Res 2006, (34 Database):D235–242. 10.1093/nar/gkj153Google Scholar
- Kasprzyk A, Keefe D, Smedley D, London D, Spooner W, Melsopp C, Hammond M, Rocca-Serra P, Cox T, Birney E: EnsMart: a generic system for fast and flexible access to biological data. Genome Res 2004, 14(1):160–169. 10.1101/gr.1645104View ArticlePubMedPubMed CentralGoogle Scholar
- Gilbert D: Shopping in the genome market with EnsMart. Brief Bioinform 2003, 4(3):292–296. 10.1093/bib/4.3.292View ArticlePubMedGoogle Scholar
- Stevens R, Baker P, Bechhofer S, Ng G, Jacoby A, Paton NW, Goble CA, Brass A: TAMBIS: transparent access to multiple bioinformatics information sources. Bioinformatics 2000, 16(2):184–185. 10.1093/bioinformatics/16.2.184View ArticlePubMedGoogle Scholar
- Etzold T, Ulyanov A, Argos P: SRS: information retrieval system for molecular biology data banks. Methods Enzymol 1996, 266: 114–128. full_textView ArticlePubMedGoogle Scholar
- The BioMoby Consortium: Interoperability with Moby 1.0-It's better than sharing your toothbrush! Briefings in Bioinformatics 2008, 9(3):220–231. 10.1093/bib/bbn003View ArticleGoogle Scholar
- Belleau F, Nolin MA, Tourigny N, Rigault P, Morissette J: Bio2RDF: towards a mashup to build bioinformatics knowledge systems. J Biomed Inform 2008, 41(5):706–716. 10.1016/j.jbi.2008.03.004View ArticlePubMedGoogle Scholar
- Ashburner M, Ball CA, Blake JA, Botstein D, Butler H, Cherry JM, Davis AP, Dolinski K, Dwight SS, Eppig JT, et al.: Gene ontology: tool for the unification of biology. The Gene Ontology Consortium. Nat Genet 2000, 25(1):25–29. 10.1038/75556View ArticlePubMedPubMed CentralGoogle Scholar
- Huang da W, Sherman BT, Stephens R, Baseler MW, Lane HC, Lempicki RA: DAVID gene ID conversion tool. Bioinformation 2008, 2(10):428–430.View ArticlePubMedGoogle Scholar
- Berriz GF, Roth FP: The Synergizer service for translating gene, protein and other biological identifiers. Bioinformatics 2008, 24(19):2272–2273. 10.1093/bioinformatics/btn424View ArticlePubMedPubMed CentralGoogle Scholar
- Iragne F, Barre A, Goffard N, De Daruvar A: AliasServer: a web server to handle multiple aliases used to refer to proteins. Bioinformatics 2004, 20(14):2331–2332. 10.1093/bioinformatics/bth241View ArticlePubMedGoogle Scholar
- Lein ES, Hawrylycz MJ, Ao N, Ayres M, Bensinger A, Bernard A, Boe AF, Boguski MS, Brockway KS, Byrnes EJ, et al.: Genome-wide atlas of gene expression in the adult mouse brain. Nature 2007, 445(7124):168–176. 10.1038/nature05453View ArticlePubMedGoogle Scholar
- Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ: Basic local alignment search tool. J Mol Biol 1990, 215(3):403–410.View ArticlePubMedGoogle Scholar
- Hubbard TJ, Aken BL, Ayling S, Ballester B, Beal K, Bragin E, Brent S, Chen Y, Clapham P, Clarke L, et al.: Ensembl 2009. Nucleic Acids Res 2009, (37 Database):D690–697. 10.1093/nar/gkn828Google Scholar
- Albert I, Thakar J, Li S, Zhang R, Albert R: Boolean network simulations for life scientists. Source Code Biol Med 2008, 3: 16. 10.1186/1751-0473-3-16View ArticlePubMedPubMed CentralGoogle Scholar
- Meeting Report: A Workshop to Plan a Deep Catalog of Human Genetic Variation2007. [http://www.1000genomes.org/bcms/1000_genomes/Documents/1000Genomes-MeetingReport.pdf]
- Hoffmann R: A wiki for the life sciences where authorship matters. Nat Genet 2008, 40(9):1047–1051. 10.1038/ng.f.217View ArticlePubMedGoogle Scholar
- Zhang Z, Cheung KH, Townsend JP: Bringing Web 2.0 to bioinformatics. Brief Bioinform 2009, 10(1):1–10. 10.1093/bib/bbn041View ArticlePubMedPubMed CentralGoogle Scholar
- Ruttenberg A, Clark T, Bug W, Samwald M, Bodenreider O, Chen H, Doherty D, Forsberg K, Gao Y, Kashyap V, et al.: Advancing translational research with the Semantic Web. BMC Bioinformatics 2007, 8(Suppl 3):S2. 10.1186/1471-2105-8-S3-S2View ArticlePubMedPubMed CentralGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.