HBICT – Hash Based Incremental Checkpointing Tool – is a free software toolkit for compression of parallel program's checkpoints. It is designed for GNU / Linux operating system (OS).

Methods and algorithms implemented in parallel programs, are characterized by different use of an available random access memory. For example, for some iterative methods it is typical to partial changing data during the operation. The checkpoints created for such programs, may contain duplicating information. Saving in CP only the changed fragments of program's status can considerably reduce size of transferred and storable data and, thereby, reduce load of an I/O subsystem. This approach is called delta compression .

HBICT supportes following types of compression:

• incremental – provides delta compression of each CP, starting from the second, relative to the previous CP;
• differential – assume compression of all CP relative to the first;
• adaptive – original algorithm, which allows to combine advantages of incremental and differential compression;
• combined – mode, providing additional processing of delta-compressed CP by algorithm LZ77, implemented in the GZIP program.

Incremental, differential and adaptive compression modes based on FIC [1]/FsCH [2] algorithm. This algorithm does not allow to detect the removal and insertion of CP fragments, but provides an acceptable level of overhead compared with the more accurate algorithms (eg, rsync).

Instructions for configuring and installing HBICT

1. Download the latest version of HBICT (hbict-X.Y.tar.gz) and save it in folder where you want to assemble package from source code.
2. Unpack the source code, configure and assemble:

tar –xzvf hbict-X.Y.tar.gz
cd hbict-X.Y
./configure --help
./configure
make
[sudo] make install

$ ./hbict
Usage:
hbict [-nzbfhu] -c <signature> [-o <outfile>] [<infile>]
    Create a delta-compressed checkpoint.
    If the  file does not exist, then treat the
    input as the 	base checkpoint and create a 
    file. By default  = stdout,  = stdin.
    -u causes hbict to update signature with the 
    current checkpoint.
hbict [-zbfh] -a [-o outfile] [infile]
    Create delta-compressed checkpoint using automatic mode.
    All needed information is stored in a service location.
    In this mode adaptive checkpoints are created (instead
    of incremental or differential checkpoints). Automatical
    suffixes ".hbict.X" are added to each compressed file
    based on a sequence number X.
hbict -r [-o <outfile>] <infile1> [<infile2> ... <infileN>]
    Create a full checkpoint file using the base checkpoint 
    and additional delta-compressed checkpoints.
    By default  = stdout.
Options can be one of the following:
    -n, --serial-number serial number of checkpoint
    -z, --gzip          additional gzip compression
    -m, --use-mmap      use mmap for reading compressed files
    -b, --block-size    hash block size
    -f, --buf-size      input buffer size

Literature

[1] Agarwal, S. Adaptive incremental checkpointing for massively parallel systems / S. Agarwal, R. Garg, M.S. Gupta, J.E. Moreira // ICS ’04: Proceedings of the 18th Annual International Conference on Supercomputing. – New York: ACM Press, 2004. – P. 277 –  286.

[2] Kiswany, S.A. stdchk: A Checkpoint Storage System for Desktop Grid Computing / S.A. Kiswany, M. Ripeanu, S.S. Vazhkudai, A. Gharaibeh // Proc. of ICDCS 2008. – USA, DC, Washington: IEEE Computer Society, 2008. – P. 613 –  624 – ISSN 978-0-7695-3172-4.