Compressing BAM files

The datasets can be found here.

​

• Note: The Deep method requires inclusion of all FASTQs that contributed reads to the BAM.
   

• Note: There is a lot that can change in the data between a FASTQ file and the corresponding BAM file - sequences may have been be trimmed, BQSR may have been applied to quality scores, read names may have changed, reads may have been filtered out or duplicates may have been collapsed to consensus sequences just to name a few. Deep will nevertheless still work, albeit a bit less effeciently, if such changes have occurred.
   

• Note: The Deep method typically consumes 20%-50% less CPU compared to compressing the BAM and FASTQ files separately. However, it consumes significant RAM - the exact amount of RAM required varies, but as a rule of thumb it is about as much as the size of the BAM file. The RAM consumption is mostly during uncompressing.
   

Example:

​

$ genozip --deep --reference hs37d5.fa.gz myfile.bam myfile-R1.fastq.gz myfile-R2.fastq.gz 

​

$ ls -lh GFX0241869*

-rw-------+ 1 57G Feb  7  2020 myfile.bam
-rw-------+ 1 31G May  3 23:28 myfile-R1.fastq.gz
-rw-------+ 1 35G May  3 23:31 myfile-R2.fastq.gz

-rw-------+ 1 16G Jun 21 19:27 myfile.deep.genozip

​

• Reducing RAM consumption: Uncompression of a .deep.genozip file is very RAM hungry. Usually, most of the RAM consumption is due to the matching of the QUAL data between the BAM and FASTQ files. Getting rid of the QUAL redundancy between BAM and FASTQ also usually accounts for the majority of benefit of Deep. Using Deep, but without matching QUAL, drastically reduces the RAM consumption, with the resulting file size being somewhere in between compressing the FASTQ and BAM separately, and full Deep.

​​

Example:

​​​

$ genozip --deep=no-qual --reference hs37d5.fa.gz myfile.bam myfile-R1.fq.gz myfile-R2.fq.gz 

​

--optimize : even better compression, with some caveats.

​​

While Genozip is primarily a lossless compressor, the --optimize option is provided to allow lossy compression as well. The idea of lossy compression is this: It is common for data to contain information in a higher resolution than we actually need for downstream analysis. Examples include resolution of base quality scores and number of decimal digits in fractions. If we could reduce the resolution, we could gain signficantly better compression. The compression-enhacing modifications to SAM/BAM/CRAM data which genozip performs when --optimize is specified are described below. They are designed to have negligible impact on downstream analysis in many common cases, however you should definitely validate this for your own specific use case.

​

Base quality scores binning: base quality scores ∈[0,93] (which appear in textual SAM files as the ASCII characters '!' through '~') are binned, loosely following Illumina's method. Binning is applied to the main QUAL field, as well as to the following tags which also contain base quality scores: QT:Z CY:Z BZ:Z and in the case of files generated by 10xGenomics or STARsolo, also to the following tags: UY:Z QX:Z sQ:Z 2Y:Z fq:Z. Note that it is not applied to OQ:Z.

​

Old values  New value

0-2         unchanged

3-9         6

10-10       15

20-24       22

25-29       27

...

85-89       87

90-92       91

93          93

​

Note: binning is not applied if the main QUAL field is already binned to 8 or fewer values.

​

Floating-point number rounding: all floating point tags (XX:f and XX:B:f) are rounded: for BAM and CRAM, they are rounded to the 10 significant bits (this provides an accuracy of approximately 3 significant decimal digits), and for SAM, they are rounded to the 3 significant digits.

Note: when displaying optimized BAM/CRAM data in textual (SAM) format, these numbers might seem surprisely not round - this is because they are displayed in base-10 - rest assured that they are indeed round in base-2.

​

IonTerrent flow signal rounding: Relevant only to IonTorrent data: the ZM:B flow signal field is modified such that negative values are changed to zero and positives are rounded to the nearest 10.

​

Example: -20,212,427 ➔ 0,210,430

​

Note: genounzip and genozip --test verify that the data uncompresses to precisely the same data as it was after the modifications. Note that this does not test the correctness of the modifications themselves.

​

Note: --optimize can take an argument, to fine grain which fields get optimized, for example:

> genozip --optimize=QUAL,rq:f    - optimizes only the QUAL and rq:f fields, if they are optimizable

​

> genozip --optimize=^QUAL,rq:f  - optimizes all optimizable fields, except QUAL and rq:f .

​

Note: To see which fields are actually optimized use --stats (you can use this with genozip, genocat or genounzip):

​

> genozip --optimize --stats test.bam

BAM file: test.bam
BAM alignments: 100,000 (in Prim VBs: 50 in Depn VBs: 53)
Contexts: 55  Vblocks: 10 x 4.0 MB  Sections: 658
Main VBs: 8 Prim VBs: 1 Depn VBs: 1
Sorting: Sorted by POS
Aligner: dragen
Buddying: sag_type=BY_SA mate=50% saggy_near=0% prim_far=0.02%
Read name style: MGI-R8
Programs: ID: Hash Table Build;ID: DRAGEN HW build
Fields optimized: QUAL,sd:f
Genozip version: 15.0.60 github

​