Compressed-Array C++ Classes¶

zfp’s compressed arrays are C++ classes, plus C wrappers around these classes, that implement random-accessible single- and multi-dimensional floating-point arrays. Since its first release, zfp provides fixed-rate arrays, zfp::array, that support both read and write access to individual array elements. As of 1.0.0, zfp also supports read-only arrays, zfp::const_array, for data that is static or is updated only infrequently. The read-only arrays support all of zfp’s compression modes including variable-rate and lossless compression.

For fixed-rate arrays, the storage size, specified in number of bits per array element, is set by the user. Such arbitrary storage is achieved via zfp’s lossy fixed-rate compression mode, by partitioning each d-dimensional array into blocks of 4^d values and compressing each block to a fixed number of bits. The more smoothly the array values vary along each dimension, the more accurately zfp can represent them. In other words, these arrays are not suitable for representing data where adjacent elements are not correlated. Rather, the expectation is that the array represents a regularly sampled and predominantly continuous function, such as a temperature field in a physics simulation.

The rate, measured in number of bits per array element, can be specified in fractions of a bit (but see FAQs #12 and #18 for limitations). zfp supports 1D, 2D, 3D, and (as of version 1.0.0) 4D arrays. For higher-dimensional arrays, consider using an array of zfp arrays. Note that array dimensions need not be multiples of four; zfp transparently handles partial blocks on array boundaries.

Read-only arrays allow setting compression mode and parameters on construction, and can optionally be initialized with uncompressed data. These arrays do not allow updating individual array elements, though the contents of the whole array may be updated by re-compressing and overwriting the array. This may be useful in applications that decompress the whole array, perform a computation that updates its contents (e.g., a stencil operation that advances the solution of a PDE), and then compress to memory the updated array.

The C++ templated array classes are implemented entirely as header files that call the zfp C library to perform compression and decompression. These arrays cache decompressed blocks to reduce the number of compression and decompression calls. Whenever an array value is read, the corresponding block is first looked up in the cache, and if found the uncompressed value is returned. Otherwise the block is first decompressed and stored in the cache. Whenever an array element is written (whether actually modified or not), a “dirty bit” is set with its cached block to indicate that the block must be compressed back to persistent storage when evicted from the cache.

This section documents the public interface to the array classes, including base classes and member accessor classes like proxy references/pointers, iterators, and views.

The following sections are available:

Read-Write Fixed-Rate Arrays
Read-Only Variable-Rate Arrays
Caching
Serialization
References
Pointers
Iterators
Views
Codec
Index

Read-Write Fixed-Rate Arrays¶

There are eight array classes for 1D, 2D, 3D, and 4D read-write arrays, each of which can represent single- or double-precision values. Although these arrays store values in a form different from conventional single- and double-precision floating point, the user interacts with the arrays via floats and doubles.

The array classes can often serve as direct substitutes for C/C++ single- and multi-dimensional floating-point arrays and STL vectors, but have the benefit of allowing fine control over storage size. All classes below belong to the zfp namespace.

Note

Much of the compressed-array API was modified in zfp 1.0.0 to support 64-bit indexing of very large arrays. In particular, array dimensions and indices now use the size_t type instead of uint and strides use the ptrdiff_t type instead of int.

Base Class¶

class array¶: Virtual base class for common array functionality.

zfp_type array::scalar_type() const¶: Return the underlying scalar type (zfp_type) of the array.

uint array::dimensionality() const¶: Return the dimensionality (aka. rank) of the array: 1, 2, 3, or 4.

array::header array::get_header() const¶: Deprecated function as of zfp 1.0.0. See the Header section on how to construct a header.

static array *array::construct(const header &h, const void *buffer = 0, size_t buffer_size_bytes = 0)¶: Construct a compressed-array object whose scalar type, dimensions, and rate are given by the header h. Return a base class pointer upon success. The optional buffer points to compressed data that, when passed, is copied into the array. If buffer is absent, the array is default initialized with all zeroes. The optional buffer_size_bytes parameter specifies the buffer length in bytes. When passed, a comparison is made to ensure that the buffer size is at least as large as the size implied by the header. If this function fails for any reason, an exception is thrown.

Common Methods¶

The following methods are common to 1D, 2D, 3D, and 4D arrays, but are implemented in the array class specific to each dimensionality rather than in the base class.

size_t array::size() const¶: Total number of elements in array, e.g., nx × ny × nz for 3D arrays.

double array::rate() const¶: Return rate in bits per value.

double array::set_rate(double rate)¶: Set desired compression rate in bits per value. Return the closest rate supported. See FAQ #12 and FAQ #18 for discussions of the rate granularity. This method destroys the previous contents of the array.

size_t array::size_bytes(uint mask = ZFP_DATA_ALL) const¶: Return storage size of components of array data structure indicated by mask. The mask is constructed via bitwise OR of predefined constants. Available as of zfp 1.0.0.

size_t array::compressed_size() const¶: Return number of bytes of storage for the compressed data. This amount does not include the small overhead of other class members or the size of the cache. Rather, it reflects the size of the memory buffer returned by compressed_data().

void *array::compressed_data() const¶: Return pointer to compressed data for read or write access. The size of the buffer is given by compressed_size().

Note

As of zfp 1.0.0, the return value is void* rather than uchar* to simplify pointer conversion and to dispel any misconception that the compressed data needs only uchar alignment. Compressed streams are always word aligned (see :c:var:`stream_word_bits` and BIT_STREAM_WORD_TYPE).

size_t array::cache_size() const¶: Return the cache size in number of bytes.

void array::set_cache_size(size_t bytes)¶: Set minimum cache size in bytes. The actual size is always a power of two bytes and consists of at least one block. If bytes is zero, then a default cache size is used, which requires the array dimensions to be known.

void array::clear_cache() const¶: Empty cache without compressing modified cached blocks, i.e., discard any cached updates to the array.

virtual void array::flush_cache() const¶: Flush cache by compressing all modified cached blocks back to persistent storage and emptying the cache. This method should be called before writing the compressed representation of the array to disk, for instance.

void array::get(Scalar *p) const¶: Decompress entire array and store at p, for which sufficient storage must have been allocated. The uncompressed array is assumed to be contiguous (with default strides) and stored in the usual “row-major” order, i.e., with x varying faster than y, y varying faster than z, etc.

void array::set(const Scalar *p)¶: Initialize array by copying and compressing data stored at p. The uncompressed data is assumed to be stored as in the get() method. If p = 0, then the array is zero-initialized.

const_reference array::operator[](size_t index) const¶: Return const reference to scalar stored at given flat index (inspector). For a 3D array, index = x + nx * (y + ny * z).

Note

As of zfp 1.0.0, the return value is no longer Scalar but is a const reference to the corresponding array element (conceptually equivalent to const Scalar&). This API change was necessary to allow obtaining a const pointer to the element when the array itself is const qualified, e.g., const_pointer p = &a[index];.

reference array::operator[](size_t index)¶: Return proxy reference to scalar stored at given flat index (mutator). For a 3D array, index = x + nx * (y + ny * z).

iterator array::begin()¶: Return random-access mutable iterator to beginning of array.

iterator array::end()¶: Return random-access mutable iterator to end of array. As with STL iterators, the end points to a virtual element just past the last valid array element.

const_iterator array::begin() const¶

const_iterator array::cbegin() const¶: Return random-access const iterator to beginning of array.

const_iterator array::end() const¶

const_iterator array::cend() const¶: Return random-access const iterator to end of array.

Note

Const references, pointers, and iterators are available as of zfp 1.0.0.

1D, 2D, 3D, and 4D Arrays¶

Below are classes and methods specific to each array dimensionality and template scalar type (float or double). Since the classes and methods share obvious similarities regardless of dimensionality, only one generic description for all dimensionalities is provided.

Note: In the class declarations below, the class template for the scalar type is omitted for readability, e.g., class array1 is used as shorthand for template <typename Scalar> class array1. Wherever the type Scalar appears, it refers to this template argument.

class array1 : public array ¶

class array2 : public array ¶

class array3 : public array ¶

class array4 : public array ¶: This is a 1D, 2D, 3D, or 4D array that inherits basic functionality from the generic array base class. The template argument, Scalar, specifies the floating type returned for array elements. The suffixes f and d can also be appended to each class to indicate float or double type, e.g., array1f is a synonym for array1<float>.

class arrayANY : public array ¶: Fictitious class used to refer to any one of array1, array2, array3, and array4. This class is not part of the zfp API.

array1::array1()¶

array2::array2()¶

array3::array3()¶

array4::array4()¶: Default constructor. Creates an empty array whose size and rate are both zero.

Note

The default constructor is useful when the array size or rate is not known at time of construction. Before the array can become usable, however, it must be resized and its rate must be set via array::set_rate(). These two tasks can be performed in either order. Furthermore, the desired cache size should be set using array::set_cache_size(), as the default constructor creates a cache that holds only one zfp block, i.e., the minimum possible.

array1::array1(size_t n, double rate, const Scalar *p = 0, size_t cache_size = 0)¶

array2::array2(size_t nx, size_t ny, double rate, const Scalar *p = 0, size_t cache_size = 0)¶

array3::array3(size_t nx, size_t ny, size_t nz, double rate, const Scalar *p = 0, size_t cache_size = 0)¶

array4::array4(size_t nx, size_t ny, size_t nz, size_t nw, double rate, const Scalar *p = 0, size_t cache_size = 0)¶: Constructor of array with dimensions n (1D), nx × ny (2D), nx × ny × nz (3D), or nx × ny × nz × nw (4D) using rate bits per value, at least cache_size bytes of cache, and optionally initialized from flat, uncompressed array p. If cache_size is zero, a default cache size suitable for the array dimensions is chosen.

array1::array1(const array::header &h, const void *buffer = 0, size_t buffer_size_bytes = 0)¶

array2::array2(const array::header &h, const void *buffer = 0, size_t buffer_size_bytes = 0)¶

array3::array3(const array::header &h, const void *buffer = 0, size_t buffer_size_bytes = 0)¶

array4::array4(const array::header &h, const void *buffer = 0, size_t buffer_size_bytes = 0)¶: Constructor from previously serialized compressed array. The header, h, contains array metadata, while the optional buffer points to the compressed data that is to be copied to the array. The optional buffer_size_bytes parameter specifies the buffer length. If the constructor fails, an exception is thrown. See array::construct() for further details on the buffer and buffer_size_bytes parameters.

array1::array1(const array1 &a)¶

array2::array2(const array2 &a)¶

array3::array3(const array3 &a)¶

array4::array4(const array4 &a)¶: Copy constructor. Performs a deep copy.

virtual array1::~array1()¶

virtual array2::~array2()¶

virtual array3::~array3()¶

virtual array4::~array4()¶: Virtual destructor (allows for inheriting from zfp arrays).

array1 &array1::operator=(const array1 &a)¶

array2 &array2::operator=(const array2 &a)¶

array3 &array3::operator=(const array3 &a)¶

array4 &array4::operator=(const array4 &a)¶: Assignment operator. Performs a deep copy.

size_t array2::size_x() const¶

size_t array2::size_y() const¶

size_t array3::size_x() const¶

size_t array3::size_y() const¶

size_t array3::size_z() const¶

size_t array4::size_x() const¶

size_t array4::size_y() const¶

size_t array4::size_z() const¶

size_t array4::size_w() const¶: Return array dimensions.

void array1::resize(size_t n, bool clear = true)¶

void array2::resize(size_t nx, size_t ny, bool clear = true)¶

void array3::resize(size_t nx, size_t ny, size_t nz, bool clear = true)¶

void array4::resize(size_t nx, size_t ny, size_t nz, size_t nw, bool clear = true)¶: Resize the array (all previously stored data will be lost). If clear is true, then the array elements are all initialized to zero.

Note

It is often desirable (though not a requirement) to also set the cache size when resizing an array, e.g., in proportion to the array size; see array::set_cache_size(). This is particularly important when the array is default constructed, which initializes the cache size to the minimum possible of only one zfp block.

const_reference array1::operator()(size_t i) const¶

const_reference array2::operator()(size_t i, size_t j) const¶

const_reference array3::operator()(size_t i, size_t j, size_t k) const¶

const_reference array4::operator()(size_t i, size_t j, size_t k, size_t l) const¶: Return const reference to element stored at multi-dimensional index given by i, j, k, and l (inspector).

Note

As of zfp 1.0.0, the return value is no longer Scalar but is a const reference to the corresponding array element (essentially equivalent to const Scalar&). This API change was necessary to allow obtaining a const pointer to the element when the array itself is const qualified, e.g., const_pointer p = &a(i, j, k);.

reference array1::operator()(size_t i)¶

reference array2::operator()(size_t i, size_t j)¶

reference array3::operator()(size_t i, size_t j, size_t k)¶

reference array4::operator()(size_t i, size_t j, size_t k, size_t l)¶: Return proxy reference to scalar stored at multi-dimensional index given by i, j, k, and l (mutator).

Read-Only Variable-Rate Arrays¶

Read-only arrays are preferable in applications that store static data, e.g., constant tables or simulation output, or data that is updated only periodically as a whole, such as when advancing the solution of a partial differential equation. Because such updates have to be applied to the whole array, one may choose to tile large arrays into smaller zfp arrays to support finer granularity updates. Read-only arrays have the benefit of supporting all of zfp’s compression modes, most of which provide higher accuracy per bit stored than fixed-rate mode.

The read-only arrays share an API with the read-write fixed-rate arrays, with only a few differences:

All methods other than those that specify array-wide settings, such as compression mode and parameters, array dimensions, and array contents, are const qualified. There are, thus, no methods for obtaining a writeable reference, pointer, or iterator. Consequently, one may not initialize such arrays one element at a time. Rather, the user initializes the whole array by passing a pointer to uncompressed data.
Whereas the constructors for fixed-rate arrays accept a rate parameter, the read-only arrays allow specifying any compression mode and corresponding parameters (if any) via a :c:struct:`zfp_config` object.
Additional methods are available for setting and querying compression mode and parameters after construction.
Read-only arrays are templated on a block index class that encodes the bit offset to each block of data. Multiple index classes are available that trade compactness and speed of access. The default hybrid4 index represents 64-bit offsets using only 24 bits of amortized storage per block. An “implicit” index is available for fixed-rate read-only arrays, which computes rather than stores offsets to equal-sized blocks.

Note

Whereas variable-rate compression almost always improves accuracy per bit of compressed data over fixed rate, one should also weigh the storage and compute overhead associated with the block index needed for variable-rate storage. The actual storage overhead can be determined by passing ZFP_DATA_INDEX to const_array::size_bytes(). This overhead tends to be small for 3D and 4D arrays.

Array initialization may be done at construction time, by passing a pointer to uncompressed data, or via the method const_array::set(), which overwrites the contents of the whole array. This method may be called more than once to update (i.e., re-initialize) the array.

Read-only arrays support a subset of references, pointers, iterators, and views; in particular those with a const_ prefix.

Currently, not all capabilities of read-write arrays are available for read-only arrays. For example, (de)serialization and construction from a view have not yet been implemented, and there are no C bindings.

Read-only arrays derive from the array base class. Additional methods are documented below.

class const_array1 : public array ¶

class const_array2 : public array ¶

class const_array3 : public array ¶

class const_array4 : public array ¶: 1D, 2D, 3D, or 4D read-only array that inherits basic functionality from the generic array base class. The template argument, Scalar, specifies the floating type returned for array elements. The suffixes f and d can also be appended to each class to indicate float or double type, e.g., const_array1f is a synonym for const_array1<float>.

class const_array : public array ¶: Fictitious class used to denote one of the 1D, 2D, 3D, and 4D read-only array classes. This pseudo base class serves only to document the API shared among the four arrays.

const_array1::const_array1()¶

const_array2::const_array2()¶

const_array3::const_array3()¶

const_array4::const_array4()¶: Default constructor. Creates an empty array whose size is zero and whose compression mode is unspecified. The array’s cache size is initialized to the minimum possible, which can have performance implications; see this note.

const_array1::const_array1(size_t n, const zfp_config &config, const Scalar *p = 0, size_t cache_size = 0)¶

const_array2::const_array2(size_t nx, size_t ny, const zfp_config &config, const Scalar *p = 0, size_t cache_size = 0)¶

const_array3::const_array3(size_t nx, size_t ny, size_t nz, const zfp_config &config, const Scalar *p = 0, size_t cache_size = 0)¶

const_array4::const_array4(size_t nx, size_t ny, size_t nz, size_t nw, const zfp_config &config, const Scalar *p = 0, size_t cache_size = 0)¶: Constructor of array with dimensions n (1D), nx × ny (2D), nx × ny × nz (3D), or nx × ny × nz × nw (4D). The compression mode and parameters are given by config (see configuration). The array uses at least cache_size bytes of cache, and is optionally initialized from flat, uncompressed array p. If cache_size is zero, a default cache size suitable for the array dimensions is chosen.

const_array1::const_array1(const const_array1 &a)¶

const_array2::const_array2(const const_array2 &a)¶

const_array3::const_array3(const const_array3 &a)¶

const_array4::const_array4(const const_array4 &a)¶: Copy constructor. Performs a deep copy.

virtual const_array1::~const_array1()¶

virtual const_array2::~const_array2()¶

virtual const_array3::~const_array3()¶

virtual const_array4::~const_array4()¶: Virtual destructor (allows for inheritance).

const_array1 &const_array1::operator=(const const_array1 &a)¶

const_array2 &const_array2::operator=(const const_array2 &a)¶

const_array3 &const_array3::operator=(const const_array3 &a)¶

const_array4 &const_array4::operator=(const const_array4 &a)¶: Assignment operator. Performs a deep copy.

size_t const_array::size() const¶: Total number of elements in array, e.g., nx × ny × nz for 3D arrays.

size_t const_array2::size_x() const¶

size_t const_array2::size_y() const¶

size_t const_array3::size_x() const¶

size_t const_array3::size_y() const¶

size_t const_array3::size_z() const¶

size_t const_array4::size_x() const¶

size_t const_array4::size_y() const¶

size_t const_array4::size_z() const¶

size_t const_array4::size_w() const¶: Return array dimensions.

void const_array1::resize(size_t n, bool clear = true)¶

void const_array2::resize(size_t nx, size_t ny, bool clear = true)¶

void const_array3::resize(size_t nx, size_t ny, size_t nz, bool clear = true)¶

void const_array4::resize(size_t nx, size_t ny, size_t nz, size_t nw, bool clear = true)¶: Resize the array (all previously stored data will be lost). If clear is true, then the array elements are all initialized to zero. See also this note.

zfp_mode const_array::mode() const¶: Currently selected compression mode. If not yet specified, zfp_mode_null is returned.

double const_array::rate() const¶: Return rate in compressed bits per value when fixed-rate mode is enabled, else zero.

uint const_array::precision() const¶: Return precision in uncompressed bits per value when fixed-precision mode is enabled, else zero.

double const_array::accuracy() const¶: Return accuracy as absolute error tolerance when fixed-accuracy mode is enabled, else zero.

void const_array::params(uint *minbits, uint *maxbits, uint *maxprec, int *minexp) const¶: Expert mode compression parameters (available for all compression modes). Pointers may be null if the corresponding parameter is not requested.

double const_array::set_reversible()¶: Enable reversible mode. This method destroys the previous contents of the array.

double const_array::set_rate(double rate)¶: Set desired rate in compressed bits per value (enables fixed-rate mode). This method destroys the previous contents of the array. See also array::set_rate().

Note

Whereas the read-write fixed-rate arrays (zfp::array) require that block storage is word aligned, the read-only arrays (zfp::const_array) are not subject to such restrictions and therefore support finer rate granularity. For a d-dimensional const_array, the rate granularity is 4^-d bits/value, e.g., a quarter bit/value for 1D arrays.

uint const_array::set_precision(uint precision)¶: Set desired precision in uncompressed bits per value (enables fixed-precision mode). This method destroys the previous contents of the array.

double const_array::set_accuracy(double tolerance)¶: Set desired accuracy as absolute error tolerance (enables fixed-accuracy mode). This method destroys the previous contents of the array.

bool const_array::set_params(uint minbits, uint maxbits, uint maxprec, int minexp)¶: Set expert mode parameters. This method destroys the previous contents of the array. Return whether the codec supports the combination of parameters.

void const_array::set_config(const zfp_config &config)¶: Set compression mode and parameters given by config (see configuration). This is a more general method for setting compression parameters such as rate, precision, accuracy, and expert mode parameters.

size_t const_array::size_bytes(uint mask = ZFP_DATA_ALL) const¶: Return storage size of components of array data structure indicated by mask. The mask is constructed via bitwise OR of predefined constants.

size_t const_array::compressed_size() const¶: Return number of bytes of storage for the compressed data. This amount does not include the small overhead of other class members or the size of the cache. Rather, it reflects the size of the memory buffer returned by compressed_data().

void *const_array::compressed_data() const¶: Return pointer to compressed data for read or write access. The size of the buffer is given by compressed_size().

size_t const_array::cache_size() const¶: Return the cache size in number of bytes.

void const_array::set_cache_size(size_t bytes)¶: Set minimum cache size in bytes. The actual size is always a power of two bytes and consists of at least one block. If bytes is zero, then a default cache size is used, which requires the array dimensions to be known.

void const_array::clear_cache() const¶: Empty cache.

void const_array::get(Scalar *p) const¶: Decompress entire array and store at p, for which sufficient storage must have been allocated. The uncompressed array is assumed to be contiguous (with default strides) and stored in the usual “row-major” order, i.e., with x varying faster than y, y varying faster than z, etc.

void const_array::set(const Scalar *p, bool compact = true)¶: Initialize array by copying and compressing floating-point data stored at p. If p = 0, then the array is zero-initialized. The uncompressed data is assumed to be stored as in the get() method. Since the size of compressed data may not be known a priori, this method conservatively allocates enough space to hold it. If compact is true, any unused storage for compressed data is freed after initialization.

const_reference const_array1::operator()(size_t i) const¶

const_reference const_array2::operator()(size_t i, size_t j) const¶

const_reference const_array3::operator()(size_t i, size_t j, size_t k) const¶

const_reference const_array4::operator()(size_t i, size_t j, size_t k, size_t l) const¶: Return const reference to element stored at multi-dimensional index given by i, j, k, and l (inspector).

const_reference const_array::operator[](size_t index) const¶: Return const reference to scalar stored at given flat index (inspector). For a 3D array, index = x + nx * (y + ny * z).

const_iterator const_array::begin() const¶

const_iterator const_array::cbegin() const¶: Return random-access const iterator to beginning of array.

const_iterator end() const¶

const_iterator cend() const¶: Return random-access const iterator to end of array.

Caching¶

As mentioned above, the array classes maintain a software write-back cache of at least one uncompressed block. When a block in this cache is evicted (e.g., due to a conflict), it is compressed back to permanent storage only if it was modified while stored in the cache.

The size cache to use is specified by the user and is an important parameter that needs careful consideration in order to balance the extra memory usage, performance, and quality (recall that data loss is incurred only when a block is evicted from the cache and compressed). Although the best choice varies from one application to another, we suggest allocating at least two “layers” of blocks, e.g., 2 × (nx / 4) × (ny / 4) blocks for 3D arrays, for applications that stream through the array and perform stencil computations such as gathering data from neighboring elements. This allows limiting the cache misses to compulsory ones. If the cache_size parameter provided to the constructor is set to zero bytes, then a default cache size of at least √n blocks is used, where n is the total number of blocks contained in the array.

The cache size can be set during construction, or can be set at a later time via array::set_cache_size(). Note that if cache_size = 0, then the array dimensions must have already been specified for the default size to be computed correctly. When the cache is resized, it is first flushed if not already empty. The cache can also be flushed explicitly if desired by calling array::flush_cache(). To empty the cache without compressing any cached data, call array::clear_cache(). To query the byte size of the cache, use array::cache_size().

By default, a direct-mapped cache is used with a hash function that maps block indices to cache lines. A faster but more collision prone hash can be enabled by defining the preprocessor macro ZFP_WITH_CACHE_FAST_HASH. A two-way skew-associative cache is enabled by defining the preprocessor macro ZFP_WITH_CACHE_TWOWAY.

Serialization¶

zfp’s read-write compressed arrays can be serialized to sequential, contiguous storage and later recovered back into an object, e.g., to support I/O of compressed-array objects. Two pieces of information are needed to describe a zfp array: the raw compressed data, obtained via array::compressed_data() and array::compressed_size(), and a header that describes the array scalar type, dimensions, and rate. The user may concatenate the header and compressed data to form a fixed-rate byte stream that can be read by the zfp command-line tool. When serializing the array, the user should first call array::flush_cache() before accessing the raw compressed data.

There are two primary ways to construct a compressed-array object from compressed data: via array-specific constructors and via a generic factory function:

When the array scalar type (i.e., float or double) and dimensionality (i.e., 1D, 2D, 3D, or 4D) are already known, the corresponding array constructor may be used. If the scalar type and dimensionality stored in the header do not match the array class, then an exception is thrown.
zfp provides a factory function that can be used when the serialized array type is unknown but described in the header. This function returns a pointer to the abstract base class, array, which the caller should dynamically cast to the corresponding derived array, e.g., by examining array::scalar_type() and array::dimensionality().

The (static) factory function is made available by including zfp/factory.hpp. This header must be included after first including the header files associated with the compressed arrays, i.e., zfp/array1.hpp, zfp/array2.hpp, zfp/array3.hpp, and zfp/array4.hpp. Only those arrays whose header files are included can be constructed by the factory function. This design decouples the array classes so that they may be included independently, for example, to reduce compilation time.

Both types of deserialization functions accept an array::header, an optional buffer holding compressed data, and an optional buffer size. If this buffer is provided, then a separate copy of the compressed data it holds is made, which is used to initialize the array. If the optional buffer size is also provided, then these functions throw an exception if the size is not at least as large as is expected from the metadata stored in the header. This safeguard is implemented to avoid accessing memory beyond the end of the buffer. If no buffer is provided, then all array elements are default initialized to zero. The array may later be initialized by directly reading/copying data into the space pointed to by array::compressed_data() and calling array::clear_cache() (in either order).

Below is a simple example of serialization of a 3D compressed array of doubles (error checking has been omitted for clarity):

zfp::array3d a(nx, ny, nz, rate);
...
a.flush_cache();
zfp::array::header h(a);
fwrite(h.data(), h.size_bytes(), 1, file);
fwrite(a.compressed_data(), a.compressed_size(), 1, file);

We may then deserialize this array using the factory function. The following example reads the compressed data directly into the array without making a copy:

zfp::array::header h;
fread(h.data(), h.size_bytes(), 1, file);
zfp::array* p = zfp::array::construct(h);
fread(p->compressed_data(), p->compressed_size(), 1, file);
assert(p->dimensionality() == 3 && p->scalar_type() == zfp_type_double);
zfp::array3d& a = *dynamic_cast<zfp::array3d*>(p);

When the array is no longer in use, call delete p; to deallocate it.

Note

The array serialization API changed significantly in zfp 1.0.0. The array::get_header() function is now deprecated and has been replaced with a header constructor that takes an array as parameter. Exceptions are now part of the main zfp namespace rather than nested within the array header. The header is no longer a simple POD data structure but should be queried for its data pointer and size.

Header¶

Short 12-byte headers are used to describe array metadata and compression parameters when serializing a compressed array. This header is the same as supported by the zfp_read_header() and zfp_write_header() functions, using ZFP_HEADER_FULL to indicate that complete metadata is to be stored in the header. The header is also compatible with the zfp command-line tool. Processing of the header may result in an exception being thrown.

Note

Compressed-array headers use zfp’s most concise representation of only 96 bits. Such short headers support compressed blocks up to 2048 bits long. This implies that the highest rate for 3D arrays is 2048/4³ = 32 bits/value; the highest rate for 4D arrays is only 2048/4⁴ = 8 bits/value. 3D and 4D arrays whose rate exceeds these limits cannot be serialized and result in an exception being thrown. 1D and 2D arrays support rates up to 512 and 128 bits/value, respectively, which both are large enough to represent all usable rates.

class array::header¶: The header stores information such as scalar type, array dimensions, and compression parameters such as rate. Compressed-array headers are always 96 bits long.

header::header()¶: Default constructor for header.

header::header(const array &a)¶: Construct header for compressed-array a. Throws an exception upon failure.

header::header(const void *buffer, size_t bytes = 0)¶: Deserialize header from memory buffer given by buffer of optional size bytes. This memory buffer is obtained from an existing header during serialization via header::data() and header::size_bytes(). The constructor throws an exception upon failure.

zfp_type header::scalar_type() const¶: Scalar type associated with array (see array::scalar_type()).

uint header::dimensionality() const¶: Dimensionality associated with array (see array::dimensionality()).

size_t header::size_x() const¶

size_t header::size_y() const¶

size_t header::size_z() const¶

size_t header::size_w() const¶: Array dimensions. Unused dimensions have a size of zero.

double header::rate() const¶: Rate in bits per value (see array::rate());

virtual const void *header::data() const = 0¶: Return pointer to header data.

virtual size_t header::size_bytes(uint mask = ZFP_DATA_HEADER) const = 0¶: When mask = ZFP_DATA_HEADER, return header payload size in bytes pointed to by header::data(). Only those bytes are needed to (de)serialize a header. The header object stores additional (redundant) metadata whose size can be queried via ZFP_DATA_META.

Exceptions¶

class exception : public std::runtime_error¶: Compressed arrays may throw this exception upon serialization, when constructing a header via its constructor, or deserialization, when constructing a compressed array via its constructor or factory function. The exception::what() method returns a std::string error message that indicates the cause of the exception. Most error messages changed in zfp 1.0.0.

References¶

class array1::const_reference¶

class array2::const_reference¶

class array3::const_reference¶

class array4::const_reference¶

class array1::reference : public array1::const_reference ¶

class array2::reference : public array2::const_reference ¶

class array3::reference : public array3::const_reference ¶

class array4::reference : public array4::const_reference ¶

Array indexing operators must return lvalue references that alias array elements and serve as vehicles for assigning values to those elements. Unfortunately, zfp cannot simply return a standard C++ reference (e.g., float&) to an uncompressed array element since the element in question may exist only in compressed form or as a transient cached entry that may be invalidated (evicted) at any point.

To address this, zfp provides proxies for references and pointers that act much like regular references and pointers, but which refer to elements by array and index rather than by memory address. When assigning to an array element through such a proxy reference or pointer, the corresponding element is decompressed to cache (if not already cached) and immediately updated.

zfp references may be freely passed to other functions and they remain valid during the lifetime of the corresponding array element. One may also take the address of a reference, which yields a proxy pointer. When a reference appears as an rvalue in an expression, it is implicitly converted to a value.

zfp 1.0.0 adds const qualified versions of references, pointers, and iterators to support const correctness and potential performance improvements when only read access is needed. As with STL containers, the corresponding types are prefixed by const_, e.g., const_reference. The mutable versions of these classes inherit the read-only API from the corresponding const versions.

Only references into read-write arrays are discussed here; the read-only arrays support the same const_reference API.

Note

Do not confuse const_reference and const reference. The former is a reference to an immutable array element, while the latter means that the proxy reference object itself is immutable.

References define a single type:

type reference::value_type¶

type const_reference::value_type¶: Scalar type associated with referenced array elements.

The following operators are defined for zfp references. They act on the referenced array element in the same manner as operators defined for conventional C++ references. References are obtained via array inspectors and mutators.

value_type reference::operator value_type() const¶

value_type const_reference::operator value_type() const¶: Conversion operator for dereferencing the reference. Return the value of the referenced array element.

pointer reference::operator&() const¶

const_pointer const_reference::operator&() const¶: Return (const) pointer to the referenced array element.

reference reference::operator=(const reference &ref)¶: Assignment (copy) operator. The referenced element, elem, is assigned the value stored at the element referenced by ref. Return *this.

reference reference::operator=(Scalar val)¶

reference reference::operator+=(Scalar val)¶

reference reference::operator-=(Scalar val)¶

reference reference::operator*=(Scalar val)¶

reference reference::operator/=(Scalar val)¶: Assignment and compound assignment operators. For a given operator op, update the referenced element, elem, via elem op val. Return *this.

Pointers¶

class array1::const_pointer¶

class array2::const_pointer¶

class array3::const_pointer¶

class array4::const_pointer¶

class array1::pointer : public array1::const_pointer ¶

class array2::pointer : public array2::const_pointer ¶

class array3::pointer : public array3::const_pointer ¶

class array4::pointer : public array4::const_pointer ¶

Similar to references, zfp supports proxy pointers (also known as fancy pointers) to individual array elements. From the user’s perspective, such pointers behave much like regular pointers to uncompressed data, e.g., instead of

float a[ny][nx];     // uncompressed 2D array of floats
float* p = &a[0][0]; // point to first array element
p[nx] = 1;           // set a[1][0] = 1
*++p = 2;            // set a[0][1] = 2

one would write

zfp::array2<float> a(nx, ny, rate);       // compressed 2D array of floats
zfp::array2<float>::pointer p = &a(0, 0); // point to first array element
p[nx] = 1;                                // set a(0, 1) = 1
*++p = 2;                                 // set a(1, 0) = 2

However, even though zfp’s proxy pointers point to individual scalars, they are associated with the array that those scalars are stored in, including the array’s dimensionality. Pointers into arrays of different dimensionality have incompatible type. Moreover, pointers to elements in different arrays are incompatible. For example, one cannot take the difference between pointers into two different arrays.

Unlike zfp’s proxy references, its proxy pointers support traversing arrays using conventional pointer arithmetic. In particular, unlike the iterators below, zfp’s pointers are oblivious to the fact that the compressed arrays are partitioned into blocks, and the pointers traverse arrays element by element as though the arrays were flattened to one-dimensional arrays. That is, if p points to the first element of a 3D array a(nx, ny, nz), then a(i, j, k) == p[i + nx * (j + ny * k)]. In other words, pointer indexing follows the same order as flat array indexing (see array::operator[]()).

A pointer remains valid during the lifetime of the array into which it points. Like conventional pointers, proxy pointers can be passed to other functions and manipulated there, for instance, by passing the pointer by reference via pointer&.

As of zfp 1.0.0, const qualified pointers const_pointer are available, and conceptually are equivalent to const Scalar*. Pointers are available for read-only arrays also.

The following operators are defined for proxy pointers. Below p refers to the pointer being acted upon.

pointer pointer::operator=(const pointer &q)¶

const_pointer const_pointer::operator=(const const_pointer &q)¶: Assignment operator. Assigns q to p.

reference pointer::operator*() const¶

const_reference const_pointer::operator*() const¶: Dereference operator. Return proxy (const) reference to the value pointed to by p.

reference pointer::operator[](ptrdiff_t d) const¶

const_reference const_pointer::operator[](ptrdiff_t d) const¶: Offset dereference operator. Return proxy (const) reference to the value stored at p[d].

pointer pointer::operator+(ptrdiff_t d) const¶

const_pointer const_pointer::operator+(ptrdiff_t d) const¶: Return a copy of the pointer incremented by d.

pointer pointer::operator-(ptrdiff_t d) const¶

const_pointer const_pointer::operator-(ptrdiff_t d) const¶: Return a copy of the pointer decremented by d.

ptrdiff_t pointer::operator-(const pointer &q) const¶

ptrdiff_t const_pointer::operator-(const const_pointer &q) const¶: Return difference p - q. Defined only for pointers within the same array.

bool pointer::operator==(const pointer &q) const¶

bool const_pointer::operator==(const const_pointer &q) const¶: Return true if p and q point to the same array element.

bool pointer::operator!=(const pointer &q) const¶

bool const_pointer::operator!=(const const_pointer &q) const¶: Return true if p and q do not point to the same array element. This operator returns false if p and q do not point into the same array.

bool pointer::operator<=(const pointer &q) const¶

bool pointer::operator>=(const pointer &q) const¶

bool pointer::operator<(const pointer &q) const¶

bool pointer::operator>(const pointer &q) const¶

bool const_pointer::operator<=(const const_pointer &q) const¶

bool const_pointer::operator>=(const const_pointer &q) const¶

bool const_pointer::operator<(const const_pointer &q) const¶

bool const_pointer::operator>(const const_pointer &q) const¶: Return true if the two pointers satisfy the given relationship. These operators return false if p and q do not point into the same array.

pointer &pointer::operator++()¶

const_pointer &const_pointer::operator++()¶: Prefix increment pointer, i.e., ++p. Return reference to the incremented pointer.

pointer &pointer::operator--()¶

const_pointer &const_pointer::operator--()¶: Prefix decrement pointer, i.e., --p. Return reference to the decremented pointer.

pointer pointer::operator++(int)¶

const_pointer const_pointer::operator++(int)¶: Postfix increment pointer, i.e., p++. Return a copy of the pointer before it was incremented.

pointer pointer::operator--(int)¶

const_pointer const_pointer::operator--(int)¶: Postfix decrement pointer, i.e., p--. Return a copy of the pointer before it was decremented.

pointer pointer::operator+=(ptrdiff_t d)¶

const_pointer const_pointer::operator+=(ptrdiff_t d)¶: Increment pointer by d. Return a copy of the incremented pointer.

pointer pointer::operator-=(ptrdiff_t d)¶

const_pointer const_pointer::operator-=(ptrdiff_t d)¶: Decrement pointer by d. Return a copy of the decremented pointer.

Iterators¶

class array1::const_iterator¶

class array2::const_iterator¶

class array3::const_iterator¶

class array4::const_iterator¶

class array1::iterator : public array1::const_iterator ¶

class array2::iterator : public array2::const_iterator ¶

class array3::iterator : public array3::const_iterator ¶

class array4::iterator : public array4::const_iterator ¶

Iterators provide a mechanism for traversing a possibly multi-dimensional array—or a view of a subset of an array—without having to track array indices or bounds. They are also the preferred mechanism, compared to nested index loops, for initializing arrays, because they sequentially visit the array one block at a time. This allows all elements of a block to be initialized together and ensures that the block is not compressed to memory before it has been fully initialized, which might otherwise result in poor compression and, consequently, larger compression errors than when the entire block is initialized as a whole. Note that the iterator traversal order differs in this respect from traversal by pointers.

Blocks are visited in raster order similarly to how individual array elements are indexed, that is, first by x, then by y, then by z, etc. Within each block, elements are visited in the same raster order. All 4^d values in a block are visited before moving on to the next block (see Fig. 1).

As of zfp 1.0.0, all iterators provided by zfp are random access iterators (previously, multi-dimensional array iterators were only forward iterators). zfp iterators are STL compliant and can be used in STL algorithms that support random access iterators.

zfp 1.0.0 adds const qualified versions of iterators, given by the const_iterator class. Such iterators are available also for read-only arrays.

Per STL mandate, the iterators define several types:

type iterator::value_type¶: The scalar type associated with the array that the iterator points into.

type iterator::difference_type¶: Difference between two iterators in number of array elements.

type iterator::reference¶: The reference type associated with the iterator’s parent array class.

type iterator::pointer¶: The pointer type associated with the iterator’s parent array class.

type iterator::iterator_category¶: Type of iterator: std::random_access_iterator_tag.

For const iterators, the following additional types are defined:

type const_iterator::const_reference¶: The immutable reference type associated with the iterator’s container class.

type const_iterator::const_pointer¶: The immutable pointer type associated with the iterator’s container class.

The following operations are defined on iterators:

iterator iterator::operator=(const iterator &it)¶

const_iterator const_iterator::operator=(const const_iterator &it)¶: Assignment (copy) operator. Make the iterator point to the same element as it.

reference iterator::operator*() const¶

const_reference const_iterator::operator*() const¶: Dereference operator. Return (const) reference to the value pointed to by the iterator.

reference iterator::operator[](difference_type d) const¶

const_reference const_iterator::operator[](difference_type d) const¶: Offset dereference operator. Return (const) reference to the value d elements relative to the current element in the iteration sequence (d may be negative). This operator executes in constant time regardless of array dimensionality but is more costly than sequential iteration via iterator::operator++().

iterator iterator::operator+(difference_type d) const¶

const_iterator const_iterator::operator+(difference_type d) const¶: Return a new iterator that has been incremented by d.

iterator iterator::operator-(difference_type d) const¶

const_iterator const_iterator::operator-(difference_type d) const¶: Return a new iterator that has been decremented by d.

difference_type iterator::operator-(const iterator &it) const¶

difference_type const_iterator::operator-(const const_iterator &it) const¶: Return difference between this iterator and it in number of elements. The difference p − q between two iterators, p and q, is negative if p < q. The iterators must refer to elements in the same array.

bool iterator::operator==(const iterator &it) const¶

bool const_iterator::operator==(const const_iterator &it) const¶: Return true if the two iterators point to the same element.

bool iterator::operator!=(const iterator &it) const¶

bool const_iterator::operator!=(const const_iterator &it) const¶: Return true if the two iterators do not point to the same element.

bool iterator::operator<=(const iterator &it) const¶

bool iterator::operator>=(const iterator &it) const¶

bool iterator::operator<(const iterator &it) const¶

bool iterator::operator>(const iterator &it) const¶

bool const_iterator::operator<=(const const_iterator &it) const¶

bool const_iterator::operator>=(const const_iterator &it) const¶

bool const_iterator::operator<(const const_iterator &it) const¶

bool const_iterator::operator>(const const_iterator &it) const¶: Return true if the two iterators satisfy the given relationship. For two iterators, p and q, within the same array, p < q if and only if q can be reached by incrementing p one or more times.

iterator &iterator::operator++()¶

const_iterator &const_iterator::operator++()¶: Prefix increment (++it). Return a reference to the incremented iterator.

iterator iterator::operator++(int)¶

const_iterator const_iterator::operator++(int)¶: Postfix increment (it++). Return the value of the iterator before being incremented.

iterator &iterator::operator--()¶

const_iterator &const_iterator::operator--()¶: Prefix decrement (--it). Return a reference to the decremented iterator.

iterator iterator::operator--(int)¶

const_iterator const_iterator::operator--(int)¶: Postfix decrement (it--). Return the value of the iterator before being decremented.

iterator iterator::operator+=(difference_type d)¶

const_iterator const_iterator::operator+=(difference_type d)¶: Increment iterator d times. Return value of incremented iterator. Although :cpp:expr:`++it` and :cpp:expr:`it += 1` are semantically equivalent, the former is more efficient for multidimensional arrays.

iterator iterator::operator-=(difference_type d)¶

const_iterator const_iterator::operator-=(difference_type d)¶: Decrement iterator d times. Return value of decremented iterator. Although :cpp:expr:`--it` and :cpp:expr:`it -= 1` are semantically equivalent, the former is more efficient for multidimensional arrays.

size_t iterator::i() const¶

size_t iterator::j() const¶

size_t iterator::k() const¶

size_t iterator::l() const¶

size_t const_iterator::i() const¶

size_t const_iterator::j() const¶

size_t const_iterator::k() const¶

size_t const_iterator::l() const¶: Return array index or local view index of element pointed to by the iterator. iterator::i() is defined for all arrays. iterator::j() is defined only for 2D, 3D, and 4D arrays. iterator::k() is defined only for 3D and 4D arrays. iterator::l() is defined only for 4D arrays.

Views¶

zfp 0.5.4 adds array views. Much like how references allow indirect access to single array elements, views provide indirect access to whole arrays, or more generally to rectangular subsets of arrays. A view of an array does not allocate any storage for the array elements. Rather, the view accesses shared storage managed by the underlying array. This allows for multiple entries into an array without the need for expensive deep copies. In a sense, views can be thought of as shallow copies of arrays.

When a view exposes a whole array array<type>, it provides similar functionality to a C++ reference array<type>& or pointer array<type>* to the array. However, views are more general in that they also allow restricting access to a user-specified subset of the array, and unlike pointers also provide for the same syntax when accessing the array, e.g., array_view(i, j) instead of (*array_ptr)(i, j).

zfp’s nested views further provide for multidimensional array access analogous to the C/C++ nested array syntax array[i][j]. Finally, zfp’s private views can be used to ensure thread-safe access to its compressed arrays.

Access to array elements through a view is via inspectors and mutators that return a const_reference or reference, respectively (see References). As of zfp 1.0.0, it is also possible to obtain pointers to array elements through views and to iterate over them. View pointers and iterators allow referencing only the elements visible through the view, e.g., a rectangular subset of an array (see Fig. 1). Those elements are indexed as if the view were a contiguous array, and pointer arithmetic assumes that the possibly smaller view and not the underlying array is flattened. Private views maintain their own cache and therefore implement their own proxy references, pointers, and iterators.

Fig. 1 An 11 × 9 element view of a 2D array of dimensions 16 × 12. The numbered elements indicate the order in which the view is traversed using pointers and iterators. We have view(10, 7) == (&view(0, 0))[87] == view.begin()[97] == view.end()[-2].

With the zfp 1.0.0 release of read-only arrays, such arrays also support the two kinds of immutable views (const_view and private_const_view). The documentation below applies to views into read-only arrays as well.

Note

Like iterators and proxy references and pointers, a view is valid only during the lifetime of the array that it references. No reference counting is done to keep the array alive. It is up to the user to ensure that the referenced array object is valid when accessed through a view.

There are several types of views distinguished by these attributes:

Read-only vs. read-write access.
Shared vs. private access.
Flat vs. nested indexing.

Each of these attributes is discussed in detail below in these sections:

Immutable view
Mutable view
Flat view
Nested view
Slicing
Private immutable view
Private mutable view

Immutable view¶

The most basic view is the immutable const_view, which supports read-only access to the array elements it references. This view serves primarily as a base class for more specialized views. Its constructors allow establishing access to a whole array or to a rectangular subset of an array. Note that like references, pointers, and iterators, views are types nested within the arrays that they reference.

class array1::const_view¶

class array2::const_view¶

class array3::const_view¶

class array4::const_view¶: Immutable view into 1D, 2D, 3D, and 4D array.

array1::const_view::const_view(array1 *array)¶

array2::const_view::const_view(array2 *array)¶

array3::const_view::const_view(array3 *array)¶

array4::const_view::const_view(array4 *array)¶: Constructor for read-only access to a whole array. As already mentioned, these views are valid only during the lifetime of the underlying array object.

array1::const_view::const_view(array1 *array, size_t x, size_t nx)¶

array2::const_view::const_view(array2 *array, size_t x, size_t y, size_t nx, size_t ny)¶

array3::const_view::const_view(array3 *array, size_t x, size_t y, size_t z, size_t nx, size_t ny, size_t nz)¶

array4::const_view::const_view(array4 *array, size_t x, size_t y, size_t z, size_t w, size_t nx, size_t ny, size_t nz, size_t nw)¶: Constructors for read-only access to a rectangular subset of an array. The subset is specified by an offset, e.g., (x, y, z) for a 3D array, and dimensions, e.g., (nx, ny, nz) for a 3D array. The rectangle must fit within the surrounding array.

size_t array1::const_view::global_x(size_t i) const¶

size_t array2::const_view::global_x(size_t i) const¶

size_t array2::const_view::global_y(size_t j) const¶

size_t array3::const_view::global_x(size_t i) const¶

size_t array3::const_view::global_y(size_t j) const¶

size_t array3::const_view::global_z(size_t k) const¶

size_t array4::const_view::global_x(size_t i) const¶

size_t array4::const_view::global_y(size_t j) const¶

size_t array4::const_view::global_z(size_t k) const¶

size_t array4::const_view::global_w(size_t l) const¶: Return global array index associated with local view index. For instance, if a 1D view has been constructed with offset x, then global_x(i) returns x + i.

size_t array1::const_view::size_x() const¶

size_t array2::const_view::size_x() const¶

size_t array2::const_view::size_y() const¶

size_t array3::const_view::size_x() const¶

size_t array3::const_view::size_y() const¶

size_t array3::const_view::size_z() const¶

size_t array4::const_view::size_x() const¶

size_t array4::const_view::size_y() const¶

size_t array4::const_view::size_z() const¶

size_t array4::const_view::size_w() const¶: Return dimensions of view.

const_reference array1::const_view::operator()(size_t i) const¶

const_reference array2::const_view::operator()(size_t i, size_t j) const¶

const_reference array3::const_view::operator()(size_t i, size_t j, size_t k) const¶

const_reference array4::const_view::operator()(size_t i, size_t j, size_t k, size_t l) const¶: Return reference to scalar stored at multi-dimensional index given by x + i, y + j, z + k, and w + l, where x, y, z, and w specify the offset into the array.

const_reference array1::const_view::operator[](size_t index) const¶: Alternative inspector for 1D arrays identical to array1::const_view::operator()().

array1::const_view::const_iterator array1::const_view::begin() const¶

array2::const_view::const_iterator array2::const_view::begin() const¶

array3::const_view::const_iterator array3::const_view::begin() const¶

array4::const_view::const_iterator array4::const_view::begin() const¶

array1::const_view::const_iterator array1::const_view::cbegin() const¶

array2::const_view::const_iterator array2::const_view::cbegin() const¶

array3::const_view::const_iterator array3::const_view::cbegin() const¶

array4::const_view::const_iterator array4::const_view::cbegin() const¶: Random-access const iterator to first element of view.

array1::const_view::const_iterator array1::const_view::end() const¶

array2::const_view::const_iterator array2::const_view::end() const¶

array3::const_view::const_iterator array3::const_view::end() const¶

array4::const_view::const_iterator array4::const_view::end() const¶

array1::const_view::const_iterator array1::const_view::cend() const¶

array2::const_view::const_iterator array2::const_view::cend() const¶

array3::const_view::const_iterator array3::const_view::cend() const¶

array4::const_view::const_iterator array4::const_view::cend() const¶: Random-access const iterator to end of view.

There are a number of common methods inherited from a base class, preview, further up the class hierarchy.

double arrayANY::const_view::rate() const¶: Return rate in bits per value. Same as array::rate().

size_t arrayANY::const_view::size() const¶: Total number of elements in view, e.g., nx × ny × nz for 3D views.

With the above definitions, the following example shows how a 2D view is constructed and accessed:

zfp::array2d a(200, 100, rate); // define 200x100 array of doubles
zfp::array2d::const_view v(&a, 10, 5, 20, 20); // v is a 20x20 view into array a
assert(v(2, 1) == a(12, 6)); // v(2, 1) == a(10 + 2, 5 + 1) == a(12, 6)
assert(v.size() == 400); // 20x20 == 400

Mutable view¶

The basic mutable view derives from the const_view but adds operators for write-access. Its constructors are similar to those for the const_view.

class array1::view : public array1::const_view ¶

class array2::view : public array2::const_view ¶

class array3::view : public array3::const_view ¶

class array4::view : public array4::const_view ¶: Mutable view into 1D, 2D, 3D, and 4D array.

array1::view::view(array1 *array)¶

array2::view::view(array2 *array)¶

array3::view::view(array3 *array)¶

array4::view::view(array4 *array)¶

array1::view::view(array1 *array, size_t x, size_t nx)¶

array2::view::view(array2 *array, size_t x, size_t y, size_t nx, size_t ny)¶

array3::view::view(array3 *array, size_t x, size_t y, size_t z, size_t nx, size_t ny, size_t nz)¶

array4::view::view(array4 *array, size_t x, size_t y, size_t z, size_t w, size_t nx, size_t ny, size_t nz, size_t nw)¶: Whole-array and sub-array mutable view constructors. See const_view constructors for details.

reference array1::view::operator()(size_t i)¶

reference array2::view::operator()(size_t i, size_t j)¶

reference array3::view::operator()(size_t i, size_t j, size_t k)¶

reference array4::view::operator()(size_t i, size_t j, size_t k, size_t l)¶: These operators, whose arguments have the same meaning as in the array accessors, return proxy references to individual array elements for write access.

Flat view¶

The views discussed so far require multidimensional indexing, e.g., (i, j, k) for 3D views. Some applications prefer one-dimensional linear indexing, which is provided by the specialized flat view. For example, in a 3D view with dimensions (nx, ny, nz), a multidimensional index (i, j, k) corresponds to the flat view index

index = i + nx * (j + ny * k)

This is true regardless of the view offset (x, y, z).

The flat view derives from the mutable view and adds operator[] for flat indexing. This operator is essentially equivalent to array::operator[]() defined for 2D, 3D, and 4D arrays. Flat views also provide functions for converting between multidimensional and flat indices.

Flat views are available only for 2D, 3D, and 4D arrays. The basic mutable view, array1::view, for 1D arrays can be thought of as either a flat or a nested view.

class array2::flat_view : public array2::view ¶

class array3::flat_view : public array3::view ¶

class array4::flat_view : public array4::view ¶: Flat, mutable views for 2D, 3D, and 4D arrays.

array2::flat_view::flat_view(array2 *array)¶

array3::flat_view::flat_view(array3 *array)¶

array4::flat_view::flat_view(array4 *array)¶

array2::flat_view::flat_view(array2 *array, size_t x, size_t y, size_t nx, size_t ny)¶

array3::flat_view::flat_view(array3 *array, size_t x, size_t y, size_t z, size_t nx, size_t ny, size_t nz)¶

array4::flat_view::flat_view(array4 *array, size_t x, size_t y, size_t z, size_t w, size_t nx, size_t ny, size_t nz, size_t nw)¶: Whole-array and sub-array flat view constructors. See const_view constructors for details.

size_t array2::flat_view::index(size_t i, size_t j) const¶

size_t array3::flat_view::index(size_t i, size_t j, size_t k) const¶

size_t array4::flat_view::index(size_t i, size_t j, size_t k, size_t l) const¶: Return flat index associated with multidimensional index.

void array2::flat_view::ij(size_t &i, size_t &j, size_t index) const¶

void array3::flat_view::ijk(size_t &i, size_t &j, size_t &k, size_t index) const¶

void array4::flat_view::ijkl(size_t &i, size_t &j, size_t &k, size_t &l, size_t index) const¶: Convert flat index to multidimensional index.

const_reference array2::flat_view::operator[](size_t index) const¶

const_reference array3::flat_view::operator[](size_t index) const¶

const_reference array4::flat_view::operator[](size_t index) const¶: Return array element associated with given flat index.

reference array2::flat_view::operator[](size_t index)¶

reference array3::flat_view::operator[](size_t index)¶

reference array4::flat_view::operator[](size_t index)¶: Return reference to array element associated with given flat index.

Nested view¶

C and C++ support nested arrays (arrays of arrays), e.g., double a[10][20][30], which are usually accessed via nested indexing a[i][j][k]. Here a is a 3D array, a[i] is a 2D array, and a[i][j] is a 1D array. This 3D array can also be accessed via flat indexing, e.g.,

a[i][j][k] == (&a[0][0][0])[600 * i + 30 * j + k]

Nested views provide a mechanism to access array elements through nested indexing and to extract lower-dimensional “slices” of multidimensional arrays. Nested views are mutable.

Nested views are associated with a dimensionality. For instance, if v is a 3D nested view of a 3D array, then v[i] is a 2D nested view (of a 3D array), v[i][j] is a 1D nested view (of a 3D array), and v[i][j][k] is a (reference to a) scalar array element. Note that the order of indices is reversed when using nested indexing compared to multidimensional indexing, e.g., v(i, j, k) == v[k][j][i].

Whereas operator[] on an array object accesses an element through flat indexing, the same array can be accessed through a nested view to in effect provide nested array indexing:

zfp::array3d a(30, 20, 10, rate); // define 30x20x10 3D array
assert(a[32] == a(2, 1, 0)); // OK: flat and multidimensional indexing
assert(a[32] == a[0][1][2]); // ERROR: a does not support nested indexing
zfp::array3d::nested_view v(&a); // define a nested view of a
assert(a[32] == v[0][1][2]); // OK: v supports nested indexing
zfp::array2d b(v[5]); // define and deep copy 30x20 2D slice of a
assert(a(2, 1, 5) == b(2, 1)); // OK: multidimensional indexing

class array2::nested_view1¶: View of a 1D slice of a 2D array.

class array2::nested_view2¶: 2D view of a 2D (sub)array.

class array3::nested_view1¶: View of a 1D slice of a 3D array.

class array3::nested_view2¶: View of a 2D slice of a 3D array.

class array3::nested_view3¶: 3D view of a 3D (sub)array.

class array4::nested_view1¶: View of a 1D slice of a 4D array.

class array4::nested_view2¶: View of a 2D slice of a 4D array.

class array4::nested_view3¶: View of a 3D slice of a 4D array.

class array4::nested_view4¶: 4D view of a 4D (sub)array.

array2::nested_view2::nested_view2(array2 *array)¶

array3::nested_view3::nested_view3(array3 *array)¶

array4::nested_view4::nested_view4(array4 *array)¶

array2::nested_view2::nested_view2(array2 *array, size_t x, size_t y, size_t nx, size_t ny)¶

array3::nested_view3::nested_view3(array3 *array, size_t x, size_t y, size_t z, size_t nx, size_t ny, size_t nz)¶

array4::nested_view4::nested_view4(array4 *array, size_t x, size_t y, size_t z, size_t w, size_t nx, size_t ny, size_t nz, size_t nw)¶: Whole-array and sub-array nested view constructors. See const_view constructors for details. Lower-dimensional view constructors are not accessible to the user but are invoked when accessing views via nested indexing.

size_t array2::nested_view1::size_x() const¶

size_t array2::nested_view2::size_x() const¶

size_t array2::nested_view2::size_y() const¶

size_t array3::nested_view1::size_x() const¶

size_t array3::nested_view2::size_x() const¶

size_t array3::nested_view2::size_y() const¶

size_t array3::nested_view3::size_x() const¶

size_t array3::nested_view3::size_y() const¶

size_t array3::nested_view3::size_z() const¶

size_t array4::nested_view1::size_x() const¶

size_t array4::nested_view2::size_x() const¶

size_t array4::nested_view2::size_y() const¶

size_t array4::nested_view3::size_x() const¶

size_t array4::nested_view3::size_y() const¶

size_t array4::nested_view3::size_z() const¶

size_t array4::nested_view4::size_x() const¶

size_t array4::nested_view4::size_y() const¶

size_t array4::nested_view4::size_z() const¶

size_t array4::nested_view4::size_w() const¶: View dimensions.

array4::nested_view3 array4::nested_view4::operator[](size_t index) const¶: Return view to a 3D slice of 4D array.

array3::nested_view2 array3::nested_view3::operator[](size_t index) const¶

array4::nested_view2 array4::nested_view3::operator[](size_t index) const¶: Return view to a 2D slice of a 3D or 4D array.

array2::nested_view1 array2::nested_view2::operator[](size_t index) const¶

array3::nested_view1 array3::nested_view2::operator[](size_t index) const¶

array4::nested_view1 array4::nested_view2::operator[](size_t index) const¶: Return view to a 1D slice of a 2D, 3D, or 4D array.

const_reference array2::nested_view1::operator[](size_t index) const¶

const_reference array3::nested_view1::operator[](size_t index) const¶

const_reference array4::nested_view1::operator[](size_t index) const¶: Return scalar element of a 2D, 3D, or 4D array.

reference array2::nested_view1::operator[](size_t index)¶

reference array3::nested_view1::operator[](size_t index)¶

reference array4::nested_view1::operator[](size_t index)¶: Return reference to a scalar element of a 2D, 3D, or 4D array.

const_reference array2::nested_view1::operator()(size_t i) const¶

const_reference array2::nested_view2::operator()(size_t i, size_t j) const¶

const_reference array3::nested_view1::operator()(size_t i) const¶

const_reference array3::nested_view2::operator()(size_t i, size_t j) const¶

const_reference array3::nested_view3::operator()(size_t i, size_t j, size_t k) const¶

const_reference array4::nested_view1::operator()(size_t i) const¶

const_reference array4::nested_view2::operator()(size_t i, size_t j) const¶

const_reference array4::nested_view3::operator()(size_t i, size_t j, size_t k) const¶

const_reference array4::nested_view4::operator()(size_t i, size_t j, size_t k, size_t l) const¶: Return const reference to a scalar element of a 2D, 3D, or 4D array.

reference array2::nested_view1::operator()(size_t i)¶

reference array2::nested_view2::operator()(size_t i, size_t j)¶

reference array3::nested_view1::operator()(size_t i)¶

reference array3::nested_view2::operator()(size_t i, size_t j)¶

reference array3::nested_view3::operator()(size_t i, size_t j, size_t k)¶

reference array4::nested_view1::operator()(size_t i)¶

reference array4::nested_view2::operator()(size_t i, size_t j)¶

reference array4::nested_view3::operator()(size_t i, size_t j, size_t k)¶

reference array4::nested_view4::operator()(size_t i, size_t j, size_t k, size_t l)¶: Return reference to a scalar element of a 2D, 3D, or 4D array.

Slicing¶

Arrays can be constructed as deep copies of slices of higher-dimensional arrays, as the code example above shows (i.e., zfp::array2d b(v[5]);). Unlike views, which have reference semantics, such array slicing has value semantics. In this example, 2D array b is initialized as a (deep) copy of a slice of 3D array a via nested view v. Subsequent modifications of b have no effect on a.

Slicing is implemented as array constructors templated on views. Upon initialization, elements are copied one at a time from the view via multidimensional indexing, e.g., v(i, j, k). Note that view and array dimensionalities must match, but aside from this an array may be constructed from any view.

Slicing needs not change the dimensionality, but can be used to copy an equidimensional subset of one array to another array, as in this example:

zfp::array3d a(30, 20, 10, rate);
zfp::array3d::const_view v(&a, 1, 2, 3, 4, 5, 6);
zfp::array3d b(v);
assert(b(0, 0, 0) == a(1, 2, 3));
assert(b.size_x() == 4);
assert(b.size_y() == 5);
assert(b.size_z() == 6);

Slicing adds the following templated array constructors.

template<class View> array1::array1(const View &v)¶

template<class View> array2::array2(const View &v)¶

template<class View> array3::array3(const View &v)¶

template<class View> array4::array4(const View &v)¶: Construct array from a view via a deep copy. The view, v, must support multidimensional indexing. The rate for the constructed array is initialized to the rate of the array associated with the view. Note that the actual rate may differ if the constructed array is a lower-dimensional slice of a higher-dimensional array due to lower rate granularity (see FAQ #12). The cache size of the constructed array is set to the default size.

Private immutable view¶

zfp’s compressed arrays are in general not thread-safe. The main reason for this is that each array maintains its own cache of uncompressed blocks. Race conditions on the cache would occur unless it were locked upon each and every array access, which would have a prohibitive performance cost.

To ensure thread-safe access, zfp provides private mutable and immutable views of arrays that maintain their own private caches. The private_const_view immutable view provides read-only access to the underlying array. It is similar to a const_view in this sense, but differs in that it maintains its own private cache rather than sharing the cache owned by the array. Multiple threads may thus access the same array in parallel through their own private views.

Note

Private views do not guarantee cache coherence. If, for example, the array is modified, then already cached data in a private view is not automatically updated. It is up to the user to ensure cache coherence by flushing (compressing modified blocks) or clearing (emptying) caches when appropriate.

The cache associated with a private view can be manipulated in the same way an array’s cache can. For instance, the user may set the cache size on a per-view basis.

Unlike with private mutable views, private immutable views may freely access any element in the array visible through the view, i.e., multiple threads may read the same array element simultaneously. For an example of how to use private views for both read and write multithreaded access, see the diffusion code example.

Private views support only multidimensional indexing, i.e., they are neither flat nor nested.

class array1::private_const_view¶

class array2::private_const_view¶

class array3::private_const_view¶

class array4::private_const_view¶: Immutable views of 1D, 2D, 3D, and 4D arrays with private caches.

array1::private_const_view::private_const_view(array1 *array)¶

array2::private_const_view::private_const_view(array2 *array)¶

array3::private_const_view::private_const_view(array3 *array)¶

array4::private_const_view::private_const_view(array4 *array)¶

array1::private_const_view::private_const_view(array1 *array, size_t x, size_t nx)¶

array2::private_const_view::private_const_view(array2 *array, size_t x, size_t y, size_t nx, size_t ny)¶

array3::private_const_view::private_const_view(array3 *array, size_t x, size_t y, size_t z, size_t nx, size_t ny, size_t nz)¶

array4::private_const_view::private_const_view(array4 *array, size_t x, size_t y, size_t z, size_t w, size_t nx, size_t ny, size_t nz, size_t nw)¶: Whole-array and sub-array private immutable view constructors. See const_view constructors for details.

size_t array1::private_const_view::size_x() const¶

size_t array2::private_const_view::size_x() const¶

size_t array2::private_const_view::size_y() const¶

size_t array3::private_const_view::size_x() const¶

size_t array3::private_const_view::size_y() const¶

size_t array3::private_const_view::size_z() const¶

size_t array4::private_const_view::size_x() const¶

size_t array4::private_const_view::size_y() const¶

size_t array4::private_const_view::size_z() const¶

size_t array4::private_const_view::size_w() const¶: View dimensions.

const_reference array1::private_const_view::operator()(size_t i) const¶

const_reference array2::private_const_view::operator()(size_t i, size_t j) const¶

const_reference array3::private_const_view::operator()(size_t i, size_t j, size_t k) const¶

const_reference array4::private_const_view::operator()(size_t i, size_t j, size_t k, size_t l) const¶: Return const reference to scalar element of a 1D, 2D, 3D, or 4D array.

The following functions are common among all dimensionalities:

size_t arrayANY::private_const_view::cache_size() const¶

void arrayANY::private_const_view::set_cache_size(size_t csize)¶

void arrayANY::private_const_view::clear_cache() const¶: Cache manipulation. See Caching for details.

Private mutable view¶

The mutable private_view supports both read and write access and is backed by a private cache. Because block compression, as needed to support write access, is not an atomic operation, mutable views and multithreading imply potential race conditions on the compressed blocks stored by an array. Although locking the array or individual blocks upon compression would be a potential solution, this would either serialize compression, thus hurting performance, or add a possibly large memory overhead by maintaining a lock with each block.

Note

To avoid multiple threads simultaneously compressing the same block, private mutable views of an array must reference disjoint, block-aligned subarrays for thread-safe access. Each block of 4^d array elements must be associated with at most one private mutable view, and therefore these views must reference non-overlapping rectangular subsets that are aligned on block boundaries, except possibly for partial blocks on the array boundary. (Expert users may alternatively ensure serialization of block compression calls and cache coherence in other ways, in which case overlapping private views may be permitted.)

Aside from this requirement, the user may partition the array into disjoint views in whatever manner is suitable for the application. The private_view API supplies a very basic partitioner to facilitate this task, but may not result in optimal partitions or good load balance.

When multithreaded write access is desired, any direct accesses to the array itself (i.e., not through a view) could invoke compression. Even a read access may trigger compression if a modified block is evicted from the cache. Hence, such direct array accesses should be confined to serial code sections when private views are used.

As with private immutable views, cache coherence is not enforced. Although this is less of an issue for private mutable views due to the requirement that views may not overlap, each private mutable view overlaps an index space with the underlying array whose cache is not automatically synchronized with the view’s private cache. See the diffusion for an example of how to enforce cache coherence with mutable and immutable private views.

The private_view class inherits all public functions from private_const_view.

class array1::private_view : public array1::private_const_view ¶

class array2::private_view : public array2::private_const_view ¶

class array3::private_view : public array3::private_const_view ¶

class array4::private_view : public array4::private_const_view ¶: Mutable views of 1D, 2D, 3D, and 4D arrays with private caches.

class array1::private_view::view_reference¶

class array2::private_view::view_reference¶

class array3::private_view::view_reference¶

class array4::private_view::view_reference¶: Proxy references to array elements specialized for mutable private views.

array1::private_view::private_view(array1 *array)¶

array2::private_view::private_view(array2 *array)¶

array3::private_view::private_view(array3 *array)¶

array4::private_view::private_view(array4 *array)¶

array1::private_view::private_view(array1 *array, size_t x, size_t nx)¶

array2::private_view::private_view(array2 *array, size_t x, size_t y, size_t nx, size_t ny)¶

array3::private_view::private_view(array3 *array, size_t x, size_t y, size_t z, size_t nx, size_t ny, size_t nz)¶

array4::private_view::private_view(array4 *array, size_t x, size_t y, size_t z, size_t w, size_t nx, size_t ny, size_t nz, size_t nw)¶: Whole-array and sub-array private mutable view constructors. See const_view constructors for details.

array1::private_view::view_reference array1::private_view::operator()(size_t i) const¶

array2::private_view::view_reference array2::private_view::operator()(size_t i, size_t j) const¶

array3::private_view::view_reference array3::private_view::operator()(size_t i, size_t j, size_t k) const¶

array4::private_view::view_reference array4::private_view::operator()(size_t i, size_t j, size_t k, size_t l) const¶: Return reference to a scalar element of a 1D, 2D, 3D, or 4D array.

The following functions are common among all dimensionalities:

void arrayANY::private_view::partition(size_t index, size_t count)¶: Partition the current view into count roughly equal-size pieces along the view’s longest dimension and modify the view’s extents to match the piece indexed by index, with 0 ≤ index < count. These functions may be called multiple times, e.g., to recursively partition along different dimensions. The partitioner does not generate new views; it merely modifies the current values of the view’s offsets and dimensions. Note that this may result in empty views whose dimensions are zero, e.g., if there are more pieces than blocks along a dimension.

void arrayANY::private_view::flush_cache() const¶: Flush cache by compressing any modified blocks and emptying the cache.

Codec¶

zfp arrays are partitioned into independent blocks that are compressed and decompressed using a codec (encoder/decoder). This codec defaults to the zfp compression scheme, but can in principle be any compression scheme or number representation that represents d-dimensional blocks of 4^d values. The zfp::array and zfp::const_array classes take such a codec class as an optional template parameter.

This section documents the API that prospective codecs must support to interface with the zfp compressed-array classes. Any one codec supports a specific scalar type (e.g., float or double), denoted Scalar below, and data dimensionality (1D, 2D, 3D, or 4D). If the codec does not support a certain compression mode, it should throw an exception when the user attempts to invoke that mode. Codecs reside in the zfp::codec namespace, e.g., zfp::codec::zfp3<Scalar> is the default codec for 3D arrays.

As of zfp 1.0.0, there is in addition to the default zfp codec a “generic” codec that allows storing data in zfp arrays in “uncompressed” form using any scalar type (specified as a template parameter). This “internal” scalar type may differ from the “external” scalar type exposed to the user through the zfp::array API. For instance, the internal type may be float while the external type is double, which provides for 2:1 fixed-rate “compression” using IEEE 754 floating point.

class codec¶: Fictitious class encapsulating the codec API. This may be thought of as a base class for the classes below specialized on dimensionality.

class codec1¶

class codec2¶

class codec3¶

class codec4¶: Fictitious classes encapsulating the codec API specialized for a given data dimensionality (1D, 2D, 3D, or 4D).

codec &codec::operator=(const codec &c)¶: Assignment operator. Performs a deep copy. This method is invoked when performing a deep copy of an array.

size_t codec::buffer_size(const zfp_field *field) const¶: Maximum buffer size needed to encode the field of given scalar type and dimensions (see zfp_stream_maximum_size()). The size should be based on the current compression mode and parameters. This method is called to determine how large a buffer to allocate and pass to codec::open().

void codec::open(void *data, size_t size)¶: Open codec for (de)compression to/from buffer pointed to by data of size bytes. The caller is responsible for allocating and deallocating this buffer, whose size is given by codec::buffer_size().

void codec::close()¶: Close codec for (de)compression.

zfp_mode codec::mode() const¶: Currently selected compression mode. See :c:enum:`zfp_mode`.

double codec::rate() const¶: Rate in compressed bits/value when fixed-rate mode is selected. See zfp_stream_rate().

uint codec::precision() const¶: Precision in uncompressed bits/value when fixed-precision mode is selected. See zfp_stream_precision().

double codec::accuracy() const¶: Accuracy as absolute error tolerance when fixed-accuracy mode is selected. See zfp_stream_accuracy().

void codec::params(uint *minbits, uint *maxbits, uint *maxprec, int *minexp) const¶: Compression parameters for any compression mode. These pointer parameters may be null if only a subset of parameters is requested. See zfp_stream_params().

void codec::set_reversible()¶: Enable reversible mode.

double codec::set_rate(double rate, bool align)¶: Set desired rate in number of compressed bits/value. When align = true, blocks are word aligned, as needed for random access writes. Return the closest rate supported. See zfp_stream_set_rate().

uint codec::set_precision(uint precision)¶: Set precision in number of uncompressed bits/value. Return the actual precision selected. See zfp_stream_set_precision().

double codec::set_accuracy(double tolerance)¶: Set accuracy as absolute error tolerance. Return the closest tolerance supported. See zfp_stream_set_accuracy().

bool codec::set_params(uint minbits, uint maxbits, uint maxprec, int minexp)¶: Set expert mode parameters. Return true on success. See zfp_stream_set_params().

bool codec::set_thread_safety(bool safety)¶: Enable or disable thread safety. This function is called whenever zfp is built with OpenMP support and when the number of mutable or immutable private views of an array changes. When two or more private views of an array are accessed by separate threads, multiple blocks may be compressed or decompressed simultaneously. The codec then has to take care that there are no race conditions on the data structures (e.g., bitstream) used for (de)compression.

size_t codec::size_bytes(uint mask = ZFP_DATA_ALL) const¶: Return storage size of components of codec data structure indicated by mask. The mask is constructed via bitwise OR of predefined constants.

static size_t codec::alignment()¶: Memory alignment in number of bytes required by codec.

static const zfp_type codec::type;: Scalar type compressed by codec.

size_t codec::encode_block(bitstream_offset offset, const Scalar *block) const¶: Encode contiguous block of 4^d scalars and store at specified bit offset within compressed-data buffer. Return the number of bits of compressed storage for the block, excluding any necessary padding. This method must flush any buffered compressed data without counting any padding (e.g., for byte alignment) in the compressed size (unless the codec requires alignment of the bit offsets).

size_t codec::decode_block(bitstream_offset offset, Scalar *block) const¶: Decode contiguous block of 4^d scalars from specified bit offset within compressed-data buffer (see codec::encode_block()). Return number of bits of compressed data decoded, excluding any padding bits, i.e., the same value reported in encoding.

size_t codec1::encode_block(bitstream_offset offset, uint shape, const Scalar *block) const¶

size_t codec2::encode_block(bitstream_offset offset, uint shape, const Scalar *block) const¶

size_t codec3::encode_block(bitstream_offset offset, uint shape, const Scalar *block) const¶

size_t codec4::encode_block(bitstream_offset offset, uint shape, const Scalar *block) const¶

Encode contiguous block of data of given shape and store at specified bit offset within compressed-data buffer. Return the number of bits of compressed storage for the block (see also codec::encode_block()).

The shape is a (2 × d)-bit encoding of the size of the d-dimensional block. For each successive pair of bits s of shape, the block size in the corresponding dimension is n = 4 - s, where 0 ≤ s ≤ 3. Thus, shape = 0 implies a full block of 4^d values. The size of the fastest varying dimension is specified in the least significant bits of shape.

size_t codec1::decode_block(bitstream_offset offset, uint shape, Scalar *block) const¶

size_t codec2::decode_block(bitstream_offset offset, uint shape, Scalar *block) const¶

size_t codec3::decode_block(bitstream_offset offset, uint shape, Scalar *block) const¶

size_t codec4::decode_block(bitstream_offset offset, uint shape, Scalar *block) const¶: Decode contiguous block of data of given shape from specified bit offset within compressed-data buffer (see also codec1::encode_block()). Return number of bits of compressed data decoded, excluding any padding bits, i.e., the same value reported in encoding.

size_t codec1::encode_block_strided(bitstream_offset offset, uint shape, const Scalar *p, ptrdiff_t sx) const¶

size_t codec2::encode_block_strided(bitstream_offset offset, uint shape, const Scalar *p, ptrdiff_t sx, ptrdiff_t sy) const¶

size_t codec3::encode_block_strided(bitstream_offset offset, uint shape, const Scalar *p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz) const¶

size_t codec4::encode_block_strided(bitstream_offset offset, uint shape, const Scalar *p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw) const¶: Encode block of data stored at p with strides sx, sy, sz, and sw. See zfp_field for information on strided storage. The shape, offset, and return value are as in codec1::encode_block().

size_t codec1::decode_block_strided(bitstream_offset offset, uint shape, Scalar *p, ptrdiff_t sx) const¶

size_t codec2::decode_block_strided(bitstream_offset offset, uint shape, Scalar *p, ptrdiff_t sx, ptrdiff_t sy) const¶

size_t codec3::decode_block_strided(bitstream_offset offset, uint shape, Scalar *p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz) const¶

size_t codec4::decode_block_strided(bitstream_offset offset, uint shape, Scalar *p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw) const¶: Decode block to strided storage pointed to by p with strides sx, sy, sz, and sw. See zfp_field for information on strided storage. The shape, offset, and return value are as in codec1::decode_block().

Index¶

To support random access, zfp arrays must know where each block is stored in memory. For fixed-rate arrays, the number of compressed bits per block is constant, and the bit offset to each block can be quickly computed. For variable-rate arrays, the compressed block size is data dependent, and additional information must be stored to index the blocks. Toward this end, zfp arrays make use of an index class that reports the offset and size (in number of bits) of each block. The zfp::array and zfp::const_array classes take such an index class as a template parameter. This index class is new as of zfp 1.0.0, which introduced variable-rate arrays.

Because zfp is designed primarily for very large arrays, the bit offset may exceed 32 bits. A straightforward implementation stores the bit offset to each block as a 64-bit integer, with the block size given by the difference of consecutive offsets. However, this overhead of 64 bits/block may exceed the payload compressed data for low-dimensional arrays or in applications like visualization that may store less than one bit per value (amortized). It is therefore important to consider more compact representations of the block index.

zfp provides multiple index classes in the zfp::index namespace that balance storage size, range of representable block offsets and sizes, and speed of access:

implicit: Used for fixed-rate storage where only the fixed number of bits per block is kept. This is the default index for fixed-rate arrays.
verbatim: This and subsequent classes support variable-rate storage. A full 64-bit offset is stored per block.
hybrid4: Four consecutive offsets are encoded together. The top 32 bits of a 44-bit base offset are stored, with the 12 least significant bits of this base set to zero. Four unsigned 16-bit deltas from the base offset complete the representation. The default for variable-rate arrays, this index offers a good tradeoff between storage, offset range, and speed.
hybrid8: Eight consecutive offsets are encoded together as two 64-bit words that store the offset to the first block (the base offset) and the sizes of the first seven blocks, from which the eight offsets are derived as a prefix sum. One 64-bit word holds the 8 least significant bits of the base offset and block sizes. The other word holds another 2 (d - 1) bits for the seven block sizes plus the top 78 - 14 d bits of the base offset, where 1 ≤ d ≤ 4 is the data dimensionality.

Properties of these index classes are summarized in Table 1.

Table 1 Properties of index classes. Storage is measured in amortized bits/block; offset and size denote supported ranges in number of bits.¶
index class	variable rate	storage	offset	size	speed
implicit		0	64	64	high
verbatim	✓	64	64	64	high
hybrid4	✓	24	44	16	medium
hybrid8	✓	16	86 - 14 d	6 + 2 d	low

This section documents the API that prospective block indices must support to interface with the zfp compressed-array classes.

class index¶: Fictitious class encapsulating the index API.

index::index(size_t blocks)¶: Construct index supporting the given number of blocks.

size_t index::size_bytes(uint mask = ZFP_DATA_ALL) const¶

bitstream_size index::range() const¶: Range of bit offsets spanned by index. This equals the total number of bits of compressed-array data.

size_t index::block_size(size_t block_index) const¶: Size of compressed block in number of bits.

bitstream_offset index::block_offset(size_t block_index) const¶: Bit offset to compressed block data.

void resize(size_t blocks)¶: Resize index to accommodate requested number of blocks. Any stored index data is destroyed.

void clear()¶: Clear all data stored by index.

void flush()¶: Flush any buffered index data. This method is called after all blocks have been compressed, e.g., in array::set().

void set_block_size(size_t size)¶: Set a fixed compressed block size in number of bits for all blocks. This method is called when fixed-rate mode is selected.

void set_block_size(size_t block_index, size_t size)¶: Set compressed block size in number of bits for a single block. For variable-rate arrays, the zero-based block_index is guaranteed to increase sequentially between calls. This method throws an exception if the index cannot support the block size or offset. The user may wish to restrict the block size, e.g., by setting maxbits in expert mode, to guard against such overflow.

static bool has_variable_rate()¶: Return true if index supports variable-sized blocks.

Compressed-Array C++ Classes¶

Read-Write Fixed-Rate Arrays¶

Base Class¶

Common Methods¶

1D, 2D, 3D, and 4D Arrays¶

Read-Only Variable-Rate Arrays¶

Caching¶

Serialization¶

Header¶

Exceptions¶

References¶

Pointers¶

Iterators¶

Views¶

Immutable view¶

Mutable view¶

Flat view¶

Nested view¶

Slicing¶

Private immutable view¶

Private mutable view¶

Codec¶

Index¶

Table Of Contents

Previous topic

Next topic