Real time data compression and decompression (packing) in the PC GAMESS/Firefly
To reduce disk space requirements and increase the effective I/O throughoutput, the PC GAMESS/Firefly implements and applies multiple real-time data compression and decompression algorithms. Internally, this is called data packing. By default, the PC GAMESS/Firefly packs two-electon integrals in AO basis (which are stored in the file AOINTS), GUGA CI Hamiltonian matrix (stored in the file WORK16), GUGA CI formula tape used by MCQDPT and XMCQDPT runs (file MCQD65), half-transformed two-electron integrals (file DASORT), half-transformed two-particle density matrix (file DAFL30), as well as some other data. Usually, this implies packing of both integer indices and the corresponding floating point values. While indices are always packed by lossless algorithms (i.e. they are unpacked exactly back to their original values), this is typically not the case of the floating point data; as the efficient packing of floating point numbers is often possible only with some minor loss of their precision. Hence, calculations that apply packing produce results that are slightly less precise as compared with the results of calculations with data packing disabled. However, the parameters of various compression routines are set automatically to guarantee that these results are of the same level of accuracy; and thus, it is generally not recommended to disable data packing.
The compression algorithms used by Firefly are fast and usually does not introduce any considerable overhead. The decompression algorithms are even much faster, as they are written directly at assembler level and with really the state-of-the-art implementation. E.g., the average data decompression rate is ca. 3-5 GB/s on modern hardware. The typical data compression ratio varies from 2x to 3.5x (and even to 10x for some data). As unpacking of data is very fast and is fully transparent, this is seen as the corresponding increase of I/O throughoutput.