hdf images hdf images

This web site is no longer maintained (but will remain online).
Please see The HDF Group's new Support Portal for the latest information.

Filters

LZO Filter

Filter ID: 305

Filter Description:

Links:

http://www.oberhumer.com/opensource/lzo/
http://www.pytables.org

Contact Information:

Francesc Alted
Email: faltet at pytables dot org


BZIP2 Filter

Filter ID: 307

Filter Description:

bzip2 is a freely available, patent free, high-quality data compressor. It typically compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical compressors), whilst being around twice as fast at compression and six times faster at decompression.

Links:

http://www.bzip.org
http://www.pytables.org

Contact Information:

Francesc Alted
Email: faltet at pytables dot org


LZF Filter

Filter ID: 32000

Filter Description:

The LZF filter is an alternative DEFLATE-style compressor for HDF5 datasets, using the free LZF library by Marc Alexander Lehmann. Its main benefit over the built-in HDF5 DEFLATE filter is speed; in memory-to-memory operation as part of the filter pipeline, it typically compresses 3x-5x faster than DEFLATE, and decompresses 2x faster, while maintaining 50% to 90% of the DEFLATE compression ratio.

LZF can be used to compress any data type, and requires no compile-time or run-time configuration. HDF5 versions 1.6.5 through 1.8.3 are supported. The filter is written in C and can be included directly in C or C++ applications; it has no external dependencies. The license is 3-clause BSD (virtually unrestricted, including commercial applications).

More information, downloads, and benchmarks, are available at the http://h5py.org/lzf/.

Additional Information:

The LZF filter was developed as part of the h5py project, which implements a general-purpose interface to HDF5 from Python.

Links:

The h5py homepage: http://h5py.org
The LZF library homepage: http://home.schmorp.de/marc/liblzf.html

Contact Information:

Andrew Collette
Web: http://h5py.org


Blosc Filter

Filter ID: 32001

Filter Description:

Blosc is a high performance compressor optimized for binary data. It has been designed to compress data very fast, at the expense of achieving lesser compression ratios than, say, zlib+shuffle. It is mainly meant to not introduce a significant delay when dealing with data that is stored in high-performance I/O systems (like large RAID cabinets, or even the OS filesystem memory cache).

It uses advanced cache-efficient techniques to reduce activity on the memory bus as much as possible. It also leverages SIMD (SSE2) and multi-threading capabilities present in nowadays multi-core processors so as to accelerate the compression/decompression process to a maximum.

Links:

http://blosc.org/
http://www.pytables.org

Contact Information:

Francesc Alted
Email: faltet at pytables dot org


MAFISC Filter

Filter ID: 32002

Filter Description:

This compressing filter exploits the multidimensionality and smoothness characterizing many scientific data sets. It adaptively applies some filters to preprocess the data and uses lzma as the actual compression step. It significantly outperforms pure lzma compression on most datasets.

The software is currently under a rather unrestrictive two clause BSD style license.

Links:

http://wr.informatik.uni-hamburg.de/research/projects/icomex/mafisc

Contact Information:

Nathanael Huebbe
Email: nathanael.huebbe @ informatik dot uni-hamburg dot de


Snappy Filter

Filter ID: 32003

Filter Description:

Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.

Snappy is widely used inside Google, in everything from BigTable and MapReduce to our internal RPC systems. (Snappy has previously been referred to as .Zippy. in some presentations and the likes.)

Links:

http://google.github.io/snappy/

Contact Information:

Michael Rissi (Dectris Ltd.)
Email: michael dot rissi at dectris dot com


LZ4 Filter

Filter ID: 32004

Filter Description:

LZ4 is a very fast lossless compression algorithm, providing compression speed at 300 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speeds up and beyond 1GB/s per core, typically reaching RAM speed limits on multi-core systems. For a format description of the LZ4 compression filter in HDF5, see HDF5_LZ4.pdf.

Links:

LZ4 Algorithm:   https://github.com/nexusformat/HDF5-External-Filter-Plugins/tree/master/LZ4

LZ4 Code:

Contact Information:

Michael Rissi (Dectris Ltd.)
Email: michael dot rissi at dectris dot com


APAX

Filter ID: 32005

Appears to be no longer available


CBF

Filter ID: 32006

Filter Description:

All imgCIF/CBF compressions and decompressions, including Canonical, Packed, Packed Vesrsion 2, Byte Offset and Nibble Offset.
License Information: GPL and LGPL

Contact Information:

Herbert J. Bernstein
Email: yayahjb at gmail dot com


JPEG-XR

Filter Id: 32007

Filter Description:

Filter that allows HDF5 image datasets to be compressed or decompressed using the JPEG-XR compression method.

Links:

JPEG-XR Compression Method
JPEG-XR Filter for HDF5

Contact Information:

Marvin Albert
Email: marvin dot albert at gmail dot com


bitshuffle

Filter Id: 32008

Filter Description:

This filter shuffles data at the bit level to improve compression. CHIME uses this filter for data acquisition.

Links:

bitshuffle
CHIME

Contact Information:

Kiyoshi Masui
Email: kiyo at physics dot ubc dot ca


SPDP

Filter Id: 32009

Filter Description:

SPDP is a fast, lossless, unified compression/decompression algorithm designed for both 32-bit single-precision (float) and 64-bit double-precision (double) floating-point data. It also works on other data.

Link to the filter:

http://cs.txstate.edu/~burtscher/research/SPDP/

Contact Information:

Martin Burtscher
Email: burtscher at txstate dot edu


LPC-Rice

Filter Id: 32010

Filter Description:

LPC-Rice is a fast lossless compression codec that employs Linear Predictive Coding together with Rice coding. It supports multi-threading and SSE2 vector instructions, enabling it to exceed compression and decompression speeds of 1 GB/s.

Link to the filter:

https://sourceforge.net/projects/lpcrice/

Contact Information:

Frans van den Bergh
Email: fvdbergh at csir dot co dot za

Derick Swanepoel
Email: dswanepoel at gmail dot com


CCSDS-123

Filter Id: 32011

Filter Description:

CCSDS-123 is a multi-threaded HDF5 compression filter using the ESA CCSDS-123 implementation.

Link to the filter:

https://sourceforge.net/projects/ccsds123-hdf-filter/

Contact Information:

Frans van den Bergh
Email: fvdbergh at csir dot co dot za

Derick Swanepoel
Email: dswanepoel at gmail dot com


JPEG-LS

Filter Id: 32012

Filter Description:

JPEG-LS is a multi-threaded HDF5 compression filter using the CharLS JPEG-LS implementation.

Link to the filter:

https://sourceforge.net/projects/jpegls-hdf-filter/

Contact Information:

Frans van den Bergh
Email: fvdbergh at csir dot co dot za

Derick Swanepoel
Email: dswanepoel at gmail dot com


zfp

Filter Id: 32013

Filter Description:

zfp is a BSD licensed open source C++ library for compressed floating-point arrays that support very high throughput read and write random access. zfp was designed to achieve high compression ratios and therefore uses lossy but optionally error-bounded compression. Although bit-for-bit lossless compression is not always possible, zfp is usually accurate to within machine epsilon in near-lossless mode, and is often orders of magnitude more accurate and faster than other lossy compressors.

Link to the filter:

https://github.com/LLNL/H5Z-ZFP

For more information see: http://computation.llnl.gov/projects/floating-point-compression/

Contact Information:

Mark Miller
Email: miller86 at llnl dot gov

Peter Lindstrom
Email: pl at llnl dot gov


fpzip

Filter Id: 32014

Filter Description:

fpzip is a library for lossless or lossy compression of 2D or 3D floating-point scalar fields. Although written in C++, fpzip has a C interface. fpzip was developed by Peter Lindstrom at LLNL.

Link to the filter:

For more information see: http://computation.llnl.gov/projects/floating-point-compression/

Contact Information:

Peter Lindstrom
Email: pl at llnl dot gov


Zstandard

Filter Id: 32015

Filter Description:

Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-offs, while being backed by a very fast decoder. The Zstandard library is provided as open source software using a BSD license.

Link to the filter:

https://github.com/aparamon/HDF5Plugin-Zstandard

Contact Information:

Andrey Paramonov
Email: paramon at acdlabs dot ru


B³D

Filter Id: 32016

Filter Description:

B³D is a fast (~1 GB/s), GPU based image compression method, developed for light-microscopy applications. Alongside lossless compression, it offers a noise dependent lossy compression mode, where the loss can be tuned as a proportion of the inherent image noise (accounting for photon shot noise and camera read noise). It not only allows for fast compression during image, but can achieve compression ratios up 100.

Information:

http://www.biorxiv.org/content/early/2017/07/21/164624


SZ

Filter Id: 32017

Filter Description:

SZ is a fast and efficient error-bounded lossy compressor for floating-point data. It was developed for scientific applications producing large-scale HPC data sets. SZ supports C, Fortran, and Java and has been tested on Linux and Mac OS X.

Link to the filter:

Information
github
License

Contact Information:

Sheng Di
Email: sdi1 at anl dot gov

Franck Cappello
Email: cappello at mcs dot anl dot gov


FCIDECOMP

Filter Id: 32018

Filter Description:

FCIDECOMP is a third-party compression filter used at EUMETSAT for the compression of netCDF-4 files. It is a codec implementing JPEG-LS using CharLS used for satellite imagery.

Link to the filter:

All software and documentation can be found at this link: ftp://ftp.eumetsat.int/pub/OPS/out/test-data/Test-data-for-External-Users/MTG_FCI_L1c_Compressed-Datasets_and_Decompression-Plugin_April2017/Decompression_Plugin/

Contact Information:

Dr. Daniel Lee
Email: daniel dot lee at eumetsat dot int


- - Last modified: 01 September 2017