An algorithm of discovering signatures from DNA databases on a computer cluster

Mutations give diversity to DNA sequences, which led to the evolution of a variety of different species and a multitude of species from the same ancestor. Even though they have similar DNA sequences from a common ancestor, due to evolution, these species also have their own unique DNA sequences which may be understood as the signatures for those particular species and can be used as a way to separate the species [1, 2]. For example, DNA signatures have already been used to identify 14 types of human pathogenic yeast [3].

Signatures are defined as DNA patterns that are significantly different from other sequences and appear only once in the sequence database. Thus, the purpose of signature discovery is to find all of the signatures in a database [4]. Much research has already been conducted in signature discovery. Amin et al. integrated multiple bioinformatics tools, including CG [5] and IslandPath [6], to determine horizontally transferred, pathotype-specific signature genes as targets for specific, high-throughput molecular diagnostic tools and reverse vaccinology screens [7]. PrimerHunter can be used to select highly sensitive and specific primers for virus subtyping identification [8]. To guarantee high sensitivity and specificity, PrimerHunter selects primers such that they efficiently amplify one of the target sequences representing different isolates of the subtype of interest, and none of the non-target sequences representing isolates of closely related virus subtypes. Accurate estimates of the melting temperature of mismatches, based on a nearest-neighbor model and calculated via a fractional programming algorithm, are used in PrimerHunter to ensure the desired amplification properties. TOFI is a tool for identifying oligonucleotide fingerprints for microarray-based pathogen diagnostic assays, which combines genome comparison tools, probe design software, and sequence alignment programs [9, 10]. TOFI is typically used to design fingerprints for a single genome. An enhanced multiple-genome pipeline presented by Satya et al. allows for efficient design of microarray probes common to groups of target genomes [11]. Insignia is web-based tool for identifying genomic signatures that are perfectly conserved by all target genomes and absent from all background genomes based on databases of bacterial and viral genomic sequences, which comprise over 8300 distinct organisms [12, 13]. TOFI designs signatures for microarray-based assays, and Insignia finds unique sequence segments that can be used to design both PCR and microarray signatures. Insignia and TOFI have the ability to identify genomic signatures that are common to multiple target genomes. Insignia and TOFI perform similar computations, but Insignia can be run online and requires less computational resources. TOFI and Insignia both build consensus regions among multiple genomes through pairwise alignments between the target genomes. Insignia reports only the unique segments in the target genomes and provides an option for users to run Primer3 [14], a PCR signature design software, on these unique segments. To quickly identify signatures in target and background genomes, Insignia has to maintain a specialized database containing pre-computed matches between every pair of genomes. However, the concomitant advantage in speed comes with the limitation that users are restricted to the target and background genomes that are part of the Insignia database, with no option to use other sequences as target or background genomes. TOPSI is a tool that extends the TOFI framework to design signatures for PCR-based pathogen diagnostic assays [15]. Like Insignia, TOPSI identifies unique segments through pairwise alignments between the input genomes. However, TOPSI goes beyond identification of unique segments, and incorporates modules to design PCR signatures from the unique segments and perform extensive specificity analysis on the designed signatures. TOPSI can provide a list of PCR signatures common to all input targets without manual manipulation. CaSSiS is capable of computing comprehensive sets of sequence- and group-specific signatures that guarantee a predefined Hamming distance, the number of mismatches with non-target sequences, from collections of deeply hierarchically clustered sequences [16]. CaSSiS tries to determine perfect group-covering signatures for every target group. For groups lacking a perfect common signature, CaSSiS finds signatures with maximal group coverage within a user-defined specificity. Zheng’s algorithm uses the Hamming distance between sequences as a measuring stick for signature discovery [17]. Suppose l and d are two whole numbers. An l-pattern represents the DNA sequence with a length of l. If two l-patterns are (l,d)-similar, this means that the Hamming distance between the two l-patterns does not exceed d. Moreover, if (l,d)-similar l-patterns could not be found, the pattern is defined as a signature under the discovery condition (l,d) in the database. Zheng’s algorithm can find all of the signatures in a database as defined above. The IMUS algorithm improved upon Zheng’s algorithm to give better discovery efficiency, but requires a larger memory [18]. Based on mathematical analysis, if a discovery condition is set as (l = 24,d = 4), when discovering signatures in a uniformly distributed database with a size of 2³⁰, IMUS requires only 7.4% of the string comparisons made by Zheng’s algorithm but creates 256 times more entries in the index. CMD is designed to discover all implicit signatures from DNA databases, where implicit signatures are signatures that satisfy discovery conditions looser than a given discovery condition [19].

However, none of the above algorithms distribute the computation of the databases onto multiple computers in a cluster. To use the algorithms in such a way, additional scripts must be applied to control the distribution and collection of the databases and results. Unfortunately, many of these approaches do not provide a formal definition for their distribution strategies. Some of the approaches, for example Insignia and CaSSiS, provide strategies to distribute the computation of the databases onto multiple computers in a cluster, but the steps of distribution and collection are not automatic. Manual manipulation is necessary to use these algorithms to distribute the computation of the databases onto multiple computers in a cluster. The match pipeline in Insignia applies strategies to reduce redundancy in sub-datasets, but relies mainly on preprocessing. PTPan [20], Jellyfish [21] and DSK [22] apply different strategies to avoid the necessity of loading the whole database into memory for searches. Each of the three approaches uses secondary storage. For example, Jellyfish and DSK use hash tables to compute the k-mers for a given k. Both algorithms achieve space efficiency by keeping most of the hash tables on disk. When counting k-mers over multiple hash tables, Jellyfish would need to store the intermediate k-mer counts on disk, which requires significantly more space, and the merge phase is not parallelized. This makes the algorithm time intensive for large databases. IMUS and Zheng’s algorithm both have two disadvantages. First, these algorithms require that the entire database to be processed (including all of the data structures that were used during computation) be loaded into memory, meaning that when the amount of data exceeds the memory capacity, these algorithms are unable to complete processing and cannot be used. Second, they are both sequential algorithms, so the time necessary for larger databases is extensive. Due to these two disadvantages, neither IMUS nor Zheng’s algorithm is suitable for applications that require processing large databases. This is a particular problem with the development of Next Generation Sequencing (NGS), as the rate of creation of sequence data is increasing daily, leading also to larger databases. This renders both IMUS and Zheng’s algorithm, which are unable to process large amounts of data and require longer processing times, unsuitable for NGS data analysis.

Divide-and-conquer is a computational strategy for solving both extensive and complicated problems and processing large amounts of data. The basic thought behind this is as follows: suppose the amount of data that needs to be processed for a problem is represented by |D|. If |D| is smaller, it can be easily solved and can be solved directly. Otherwise, the problem may be divided into multiple smaller scale subproblems with close similarities to the original problem. These subproblems may be solved recursively, and the results combined to find a solution to the original problem. Therefore, with the divide-and-conquer strategy, each recursion may include three main steps: (1) solve: if the problem is smaller in scale and easy to solve, it looks for a solution directly; (2) divide-and-recur: divide the original problem into multiple smaller scale subproblems closely similar to the original problem, then recursively try to find the solution to each subproblem; (3) combine: take the solutions from the subproblems and combine to find the solution to the original problem [23]. In addition, as technology has matured, the price of multi-core CPUs has continued to fall, so the possibility to use parallel processing technology on a computer cluster to enhance processing efficiency has greatly improved. In fact, parallel processing technology is already used in many bioinformatics research fields, such as sequence alignment and analysis, protein structure prediction, and motif finding [24–31]. If we can use the divide-and-conquer strategy and parallel processing technology in signature discovery, this will improve the efficiency of discovery in large databases, which will be immensely helpful.

In this research, we propose a signature discovery algorithm called distributed divide-and-conquer-based signature discovery (DDCSD) algorithm. The DDCSD algorithm is designed specifically for discovering signatures on a computer cluster. The DDCSD algorithm automatizes the steps of distributing the database and collecting the unique signatures. The signatures are discovered from the database and provided to users without manual manipulation. The DDCSD algorithm uses the divide-and-conquer strategy to overcome the problem of processing large databases and compares multiple patterns in parallel to accelerate signature discovery. Therefore, the algorithm not only shortens the amount of time needed for discovery, it also is able to process the large databases that could not be processed in the past using IMUS and Zheng’s algorithm. In addition, by setting the threshold value of the direct discovery, DDCSD can limit the memory requirement in discovery to the memory size of the computers in the cluster. More specifically, the DDCSD algorithm can process any amount of data and is not limited by the amount of memory available. The DDCSD algorithm is implemented using a basic divide-and-conquer strategy as the basic structure. First, it decides whether to do direct discovery based on the size of the database. If the database is too large to load in its entirety, it will split the database into two equal parts and recursively processes the parts. As the recursive processing is in progress, the amount of data in a single part will gradually decrease until it can load the single part all into the memory of one computer in the cluster at one time. At the end, it will combine the results that were found separately in the two different parts and find the signatures in the original database. The DDCSD algorithm gives the formal definition in recursion for the dataset distribution strategy, that is not provided by the previous approaches. The DDCSD algorithm includes main and discovery routines. The main routine organizes discovery in a planned way. The discovery routine is used to find the unique patterns from a specified dataset in another dataset. The computation of discovery and collection in DDCSD is distributed onto discovery nodes for parallelization. Based on the experiments made on the human whole-genome EST database that has approximately 2.46G bases, the DDCSD algorithm proposed here can successfully process that database. Whereas previous algorithms could not process databases so large, the DDCSD algorithm took 1.89 hours to find all of the signatures under the discovery condition (30,2) on the cluster of ten discovery computers with 32 GB memory. The main contribution of this research is utilizing the divide-and-conquer strategy in signature discovery to process discovery in large databases, something previous algorithms were unable to do, and providing a parallel signature discovery algorithm on a cluster, that can process databases of any size regardless of the amount of memory available. This algorithm can be applied to NGS data analysis and other analysis of large databases.

Suppose that l and d represent the length and the number of allowed mismatches of signatures, respectively, and Λ is a dataset made up of l-patterns. We define signatures in Λ under a discovery condition (l,d) as patterns that exist in Λ and where there are no other (l,d)-similar patterns inside of Λ. The purpose of this research is to utilize a divide-and-conquer strategy to provide a parallel algorithm that can rapidly discover the signatures in datasets with massive amounts of data on a computer cluster.

For any subset Θ of Λ, if no (l,d)-similar pattern can be found in Θ, this pattern is considered unique in Θ. According to this definition, we can deduce that if one pattern P is a signature in Λ, then P must be unique in Θ. Therefore, if we divide Λ into two partitions of equal size (Λ _i and Λ _j ), then P will be a signature for either Λ _i or Λ _j and will be unique to the other partition. Thus, when the signatures of Λ _i and Λ _j are combined, they will include all of the signatures for Λ, making them valid candidates to discover signatures in Λ. Most importantly, no matter how many levels of recursive processing are applied, this characteristic still stands, meaning that we can use the divide-and-conquer strategy on a computer cluster to deal with the original problem posed to signature discovery algorithms where they could not process large databases. Using the above as the foundation, we designed a distributed divide-and-conquer-based signature discovery (DDCSD) algorithm that can rapidly discover the signatures that satisfy the discovery condition (l,d) in a large dataset on a computer cluster. The DDCSD algorithm includes main and discovery routines. The discovery routine accepts the candidate and source datasets that are made up of l-patterns and will find the patterns that are unique in the source in the candidate. It must be made clear that when the candidate and source are set as the same dataset, the patterns found by the discovery routine are the signatures for the dataset. Each of the computers in the cluster is called a node. The node that handles the main routine is called a main node, and those that handle the discovery routine are called discovery nodes.

The main routine of the DDCSD algorithm is shown in Figure 1. The symbols used in DDCSD are presented in Table 1. The DDCSD algorithm first examines the size of the dataset Λ; if the number of patterns is less than or equal to the preset threshold value, N, then it will automatically send Λ to a discovery node and use the discovery routine to find the signatures in Λ. The discovery result is sent back to the main node. The threshold value is decided based on the memory space of discovery nodes and is set so that the patterns in Λ (including all of the used data structures) can be loaded into the memory. On the other hand, if the number of patterns is more than the preset threshold, then Λ will be divided into two equal partitions Λ _i and Λ _j , with each being recursively processed individually. After recursive processing, the algorithm combines the results to find the signatures of Λ. Suppose that Ω _i and Ω _j represent all of the signatures in Λ _i and Λ _j , respectively. Because all signatures of Λ must be present in either Ω _i or Ω _j and will be unique to the other partition, additional comparisons must be made between Ω _i and Λ _j as well as between Ω _j and Λ _i to find the patterns unique in the other partition and delete the non-unique ones. Therefore, DDCSD uses the discovery routine to find the unique patterns in Λ _j that are in Ω _i and the unique patterns in Λ _i that are in Ω _j . These two discovery results are then combined, with the final result being the signatures of Λ.

Suppose that P and Q are two l-patterns. If P is divided into equal and non-overlapping ⌈l/γ⌉ number of γ-patterns, these γ-patterns are called γ-segments of P. P_γ,i represents the i-th γ-segment in P. P is called (γ,i,δ)-matched to a γ-pattern Γ if P_γ,i is (γ,δ)-similar to Γ. We arrive at the observation that if P and Q are (l,d)-similar, for a given γ, there will be at least one i such that P is (γ,i,⌊γ d/l⌋)-matched to Q_γ,i. Using the observation as the foundation, we designed the discovery routine of DDCSD.

The discovery routine of the DDCSD algorithm runs on discovery nodes. The discovery routine allows multiple processors to compare similarity of different patterns at the same time to allow for faster discovery speed. The discovery routine is shown in Figure 2. Suppose that the candidate and source datasets received by the discovery routine are Ξ and Θ, respectively. First of all, the discovery routine will set a suitable γ according to the memory that is available in the discovery node, where γ is a whole number between ⌈l/(d+1)⌉ and ⌈l/2⌉. The larger the number, the less strings are compared during a discovery, but more memory is needed. Conversely, the smaller the number, the more strings are compared during a discovery, but the memory requirement will be smaller. Suppose that Υ is a γ-pattern. A (Υ,γ,i,l,d)-group is a group of l-patterns. All of the l-patterns are (γ,i,⌊γ d/l⌋)-matched to Υ. According to the γ-segments included in the l-patterns in Ξ and Θ, the discovery routine assigns the l-patterns to (Υ,γ,i,l,d)-groups. More specifically, if P is in Ξ, then P will be put into (P_γ,i,γ,i,l,d)-groups, where 1 ≤ i ≤ ⌈l/γ⌉. For exampmle, assume that l, d and γ are 4, 2 and 2, respectively. If P = ’ACGT’, then P will be put into (’AC’,2,1,4,2)- and (’GT’,2,2,4,2)-groups. Assume that Δ is one of the (P_γ,i,γ,i,l,d)-groups. Δ_Ξ represents the set of Ps in Δ. According to the size of the memory, the discovery routine can pull the patterns that have yet to be processed in Θ. If there are too many patterns in Θ which cannot be loaded into the memory all at once, it can split them into multiple parts and load and process the parts one at a time. Suppose that Φ represents the group of patterns that are loaded at this time. If Q is in Φ and Q is (γ,i,⌊γ d/l⌋)-matched to a γ-pattern Γ, Q will be put into (Γ,γ,i,l,d)-groups, where 1 ≤ i ≤ ⌈l/γ⌉. For example, assume that l, d and γ are 4, 2 and 2, respectively. If Q = ’TGCA’, then Q will be put into (Γ₁,2,1,4,2)- and (Γ₂,2,2,4,2)-groups, where Γ₁ ∈ {’TG’,’GG’,’CG’,’AG’,’TT’,’TC’,’TA’} and Γ₂ ∈ {’CA’,’AA’,’GA’,’TA’,’CC’,’CG’,’CT’}. Assume that Δ is one of the (Γ,γ,i,l,d)-groups. Δ_Φ represents the set of Qs in Δ. Pairing this definition with the previous observation, we find that the patterns that are (l, d)-similar to the patterns in Δ_Ξ must be present in Δ_Φ. Therefore, for the patterns in Δ_Ξ, when examining whether they are unique, this principle can be applied to limit the discovery to similar patterns to those patterns to decrease the number of patterns compared. Each time a processor in the discovery node completes the task that it is given, the discovery routine takes a (Υ,γ,i,l,d)-group for that processor to process, which allows for parallel processing. Suppose the (Υ,γ,i,l,d)-group taken was Δ. For an l-pattern P in Δ_Ξ, when searching for (l,d)-similar patterns to P in Φ, the discovery routine only compares P and the patterns in Δ_Φ to find whether there are (l,d)-similar patterns to P. If no (l,d)-similar pattern to P is found in Δ_Φ, then it means that P is unique in Φ. Conversely, it is not unique and is deleted. The discovery routine repeats the above process until all (Υ,γ,i,l,d)-groups are processed.

In this research, we proposed a distributed divide-and-conquer-based signature discovery (DDCSD) algorithm. The DDCSD algorithm uses a divide-and-conquer strategy to overcome the problem of processing larger databases, thus solving the disadvantage of previous algorithms that could not process large databases. Also, a parallel computation mechanism on a computer cluster was used to accelerate the signature discovery. Therefore, this algorithm is not limited by the amount of memory available, and can rapidly find signatures in large databases, making it applicable to analysis of NGS and other large amounts of data.