Fast lossless compression via cascading Bloom filters

Technical background

A Bloom filter (BF) is an array A of size m having all positions initially marked 0. Elements are inserted into A by applying a collection of h hash functions: the output of each specifies a position to be marked with a 1 in A. Querying whether or not an element has been inserted involves applying the same h hash functions and checking the values at the positions they return. If at least one hash function returns 0, the element definitely was not inserted; if all 1s return, either it has been inserted, or it is a false positive. For a BF of size m, n entries can be inserted by h hash functions to achieve a false positive rate F ≈ (1 − e^(−hn/m)) ^h . In [11], it is shown that for fixed m and n, F is minimized with h = ln(2)ρ, where ρ = m/n. Plugging this value back in for F leads to F = c ^ρ , where c = 0.6185.

Encoding and decoding using a Bloom filter

Our method involves two basic processes: BF loading and querying. We initially assume all reads are unique and later relax this assumption. We load all reads into a BF B, and then use the reference genome to query it. We query B with read length subsequences (and their reverse complements) from all possible start positions on the genome. This allows us to identify all of the potential reads that correspond to genome positions, a set that covers most of the hashed reads. Some of the accepted reads will be false positives. In order to avoid them in the decoding process, we identify a set FP corresponding to all reads accepted by B that are not in the original read set. Additionally, since the reads are taken from a specimen whose genome contains mutations compared to the reference (and since sequencing is error-prone), some reads will not be recovered by querying the genome. We call this set of reads FN. FN and FP are stored separately from B, and compressed using an off-the-shelf compression tool. For a set of unique reads, this suffices to allow a complete reconstruction of the reads.

Decoding proceeds by decompressing B, FN, and FP, and then repeating the querying procedure. We initialize the read set to FN. Then, we query B with each position from the genome as done to identify elements of FP. Whenever B accepts we check if the accepted read is not also in FP, and add it to the read set if it isn't. To remove the unique read restriction, we first move all repeated reads to FN before loading B. We treat reads containing 'N' characters similarly. These two additions allow us to circumvent an inherent limitation of Bloom filters - the loss of multiplicity information - and reduces the entropy in the (now multi-) set FN, making it more compressible. The encoding process with one BF is detailed in steps 1-4 of Figure 1 and Algorithm 1. The relative contributions of error reads and repeated reads to FN are discussed in the appendix.

Algorithm 1 Encode one Input: R, G; Output: B, FN, FP Conventions: Let g be the length of the reference genome G, q _i be the i ^th genome query, ℓ _read be the sequenced read length, and P be the set of genome queries accepted by B. For brevity, we exhibit queries from only the forward strand, whereas our implementation queries (and accepts from) both strands.

FN := {r : r ∈ R and (r is repeated in R or r contains an 'N')}

R' := R \ FN

for all r ∈ R' do

insert(r, B)

end for

for all i ∈ [1, g − ℓ _read + 1] do

;if qi ∈ B then

P := P ∪ q_i

end if

end for

FN := FN ∪ {R' \ P}

FP := P \ R'

return (B, FN, FP )

Encoding and decoding using a BF cascade

Although appealingly simple, we found the above method did not offer competitive compression, as the costs imposed encoding FP and FN outweighed the benefit of storing the unique reads in B. Thus, to reduce the number of false positives that need to be compressed separately, we use a cascade of BFs as in [10]. To this end, we rename B and FP above as B₁ and FP₁, respectively. We consider B₁ to be the first BF in a cascade, and each element of FP₁ is then hashed into a subsequent BF B₂. We note that since B₂ is meant to store false positive reads it should reject true reads: thus, any element of R' (the set of unique reads) accepted by B₂ is a false positive relative to B₂. Thus, to identify FP₂, we add each element accepted by querying R' against B₂. This process can be continued for any desired number of BFs. Once BFs are loaded in this way, to identify real reads, we query each BF in the cascade and accept reads only if the index of the first BF to reject them is even.

Here the left hand side represents the sum of costs due to the expected number of elements in each BF for an infinite cascade. In practice, we use four BFs and a numerical solver in scipy [12] employing the L-BFGS-B [13] algorithm to find the value of ρ minimizing the above expression. The small list FP₄ is encoded separately along with FN. The process is described in Figure 1 steps 5-11 and Algorithm 2. Decoding proceeds using queries from G as before, but in this case each accepted read is used to query subsequent BFs until rejection. This is depicted in Figure 2and Algorithm 3.

Algorithm 2 Encoding Let B _j be the j ^th BF loaded (j ∈ [2, 4]) with FP_j-1, S ∩ B _j is short-hand notation for the subset of S accepted by B _j .

   (B₁, FN, FP₁) := Encode one(R, G) # we initialize by calling Algorithm 1

   F P₂ := R' ∩ B2

   for j = 3 to j = 4 do

      for all r ∈ FP_{j− 1}do

         insert(r, B _j )

      end for

      FP_j := FP_j−2 ∩ B_j

   end for

   return (B₁, B₂, B₃, B₄, FN, FP₄)

Algorithm 3 Decoding Input: (B₁, B₂, B₃, B₄, FN, FP₄, G); Output: (R _rc ) For brevity, reconstruction of only one strand is shown.

   R_rc := FN

   for all i ∈ [1, g − ℓ _read + 1] do

    for j = 1 to j = 4 do

      if$q_i\mathrm{\epsilon ̸}{B}_j$then

        if j is even then

          R_rc := R_rc ∪ q_i

        end if

      continue {increment i}

    end if

   end for

   if j = 4 and q _i ∈ FP₄ then

    R_rc := R_rc ∪ q_i

   end if

  end for

  return R _rc

Additional compression

BF parameters are automatically set to make each BF more compressible. This involves incrementing the number of hash functions for each BF from 1 to the minimal number that allows it to both have an uncompressed file size lower than a preset threshold (we used 500 MB) and obtain the value of F from equation 1. Typically, this results in h being in the range of 1 to 3. We do this in order to reduce each BF's compressed size (at the expense of increasing its RAM occupation); this practice is introduced in [11].

Once BFs are loaded and the sets FP 4 and FN are identified, we use 7zip [14] to compress the B 1, ..., B 4 and SCALCE [4] to compress the output lists FP₄ and FN. In principle, any general compression tool can be used for the BFs, and it is preferable to use a tool that takes advantage of existing sequence overlaps among the leftover reads to compress them efficiently.

Comparison on simulated reads

We simulated reads from Human (hg19) chromosome 20 using dwgsim [15]. This tool introduces mutations into the reference genome and then samples reads from both genome strands using a user-defined per base error rate. We sampled 100 bp single end reads at various coverage levels with a 0.001 mutation rate and a per base error rate increasing from 0 to 0.005 from the 5' to the 3' end of reads (in line with current estimates of Illumina error rates [16]). We also demonstrated the effect of varying the error rate In Figure 4. All reported results were run on a 16 core AMD Opteron 6140 (2.6 GHz) 128 GB RAM server, running the Ubuntu 12.04 Linux operating system.

Overall, we found the compression ratio improved with greater coverage for all reference-free methods, and remained essentially constant for reference-based methods. Figure 3 shows that reference-based compressors are better in compression ratios but reference-free compressors are faster (An exception to this trend was fastqz, whose reference-based version is faster than its reference-free version, likely due to the use of context model-based arithmetic coding). Quip performed poorly in compressing sequences without a reference, showing it has apparently been optimized for speed and perhaps compression of qualities and read names. SCALCE shows strong dependence of compression ratio on coverage, as would be expected by its leverage of the recurrence of long subsequences. BARCODE takes advantage of this trend to also improve with higher coverage, even as the proportion of reads hashed to BFs decreases (See Table 1). BARCODE's times are closest to SCALCE and reference-free quip, and its compression ratios approach those of reference based methods, especially at higher coverage values. For most coverage values, it maintains an order of magnitude time advantage vs. reference-based methods (~2-3x vs. fastqz, ~5-7x vs. quip), as well as an order of magnitude compression advantage of the tested reference-free methods.

Higher coverage, longer reads

Real data test

We examined BARCODE's performance on the C. Elegans data set tested in the Sequence Squeeze competition, SRR065390_1. This data set consists of 33415360 100 bp reads, amounting to 33-fold coverage of the genome. BARCODE's compression ratio on this data was 0.46 bits per base, and run time was 1203 seconds, in line with experiments described in the main text and comparable with reference-based methods tested in the Sequence Squeeze competition [1].

Contributions of repeated reads vs. errors to FN

We model the contribution of error to FN using Binom(100, p) with p = 0.0025, the mean error over the read length used in our simulated reads (where error varies from 0 to 0.005 from the 5' to 3' ends). Most of the mass is carried by the one and two error terms, leading to a relative error proportion estimate of $\left(\begin{array}{c}\hfill 100\hfill \\ \hfill 2\hfill \end{array}\right){\left(1-p\right)}^{98}{p}^2+\left(\begin{array}{c}\hfill 100\hfill \\ \hfill 2\hfill \end{array}\right){\left(1-p\right)}^{99}p$. Table 4 shows this proportion is independent of coverage level.