Blosc is a high performance compressor optimized for binary data. It
has been designed to transmit data to the processor cache faster than
the traditional, non-compressed, direct memory fetch approach via a
memcpy() OS call. Blosc is the first compressor (that I'm aware of)
that is meant not only to reduce the size of large datasets on-disk or
in-memory, but also to accelerate memory-bound computations.
It uses the blocking technique so as to reduce activity in the memory
bus as much as possible. In short, this technique works by dividing
datasets in blocks that are small enough to fit in caches of modern
processors and perform compression / decompression there. It also
leverages, if available, SIMD instructions (SSE2, AVX2) and
multi-threading capabilities of CPUs, in order to accelerate the
compression / decompression process to a maximum.
aa3ce12ced