Compressed Arrays

zfp’s compressed arrays are C++ classes, plus C wrappers around these classes, that implement random-accessible single- and multi-dimensional floating-point arrays whose storage size, specified in number of bits per array element, is set by the user. Such arbitrary storage is achieved via zfp’s lossy fixed-rate compression mode, by partitioning each d-dimensional array into blocks of 4d values and compressing each block to a fixed number of bits. The more smoothly the array values vary along each dimension, the more accurately zfp can represent them. In other words, these arrays are not suitable for representing data where adjacent elements are not correlated. Rather, the expectation is that the array represents a regularly sampled and predominantly continuous function, such as a temperature field in a physics simulation.

The rate, measured in number of bits per array element, can be specified in fractions of a bit (but see FAQs #12 and #18 for limitations). Note that array dimensions need not be multiples of four; zfp transparently handles partial blocks on array boundaries.

The C++ templated array classes are implemented entirely as header files that call the zfp C library to perform compression and decompression. These arrays cache decompressed blocks to reduce the number of compression and decompression calls. Whenever an array value is read, the corresponding block is first looked up in the cache, and if found the uncompressed value is returned. Otherwise the block is first decompressed and stored in the cache. Whenever an array element is written (whether actually modified or not), a “dirty bit” is set with its cached block to indicate that the block must be compressed back to persistent storage when evicted from the cache.

This section documents the public interface to the array classes, including base classes and member accessor classes like proxy references/pointers, iterators, and views.

The following sections are available:

Array Classes

Currently there are six array classes for 1D, 2D, and 3D arrays, each of which can represent single- or double-precision values. Although these arrays store values in a form different from conventional single- and double-precision floating point, the user interacts with the arrays via floats and doubles.

The array classes can often serve as direct substitutes for C/C++ single- and multi-dimensional floating-point arrays and STL vectors, but have the benefit of allowing fine control over storage size. All classes below belong to the zfp namespace.

Base Class

class array

Virtual base class for common array functionality.

double array::rate() const

Return rate in bits per value.

double array::set_rate(double rate)

Set desired compression rate in bits per value. Return the closest rate supported. See FAQ #12 and FAQ #18 for discussions of the rate granularity. This method destroys the previous contents of the array.

virtual void array::clear_cache() const

Empty cache without compressing modified cached blocks, i.e., discard any cached updates to the array.

virtual void array::flush_cache() const

Flush cache by compressing all modified cached blocks back to persistent storage and emptying the cache. This method should be called before writing the compressed representation of the array to disk, for instance.

size_t array::compressed_size() const

Return number of bytes of storage for the compressed data. This amount does not include the small overhead of other class members or the size of the cache. Rather, it reflects the size of the memory buffer returned by compressed_data().

uchar *array::compressed_data() const

Return pointer to compressed data for read or write access. The size of the buffer is given by compressed_size().

uint array::dimensionality() const

Return the dimensionality (1, 2, or 3) of the array.

zfp_type array::scalar_type() const

Return the underlying scalar type (zfp_type) of the array.

array::header array::get_header() const

Return a short fixed-length header describing the scalar type, dimensions, and rate associated with the array. An array::header::exception is thrown if the header cannot describe the array.

static array *array::construct(const array::header &h, const uchar *buffer = 0, size_t buffer_size_bytes = 0)

Construct a compressed-array object whose scalar type, dimensions, and rate are given by the header h. Return a pointer to the base class upon success. The optional buffer points to compressed data that, when passed, is copied into the array. If buffer is absent, the array is default initialized with all zeroes. The optional buffer_size_bytes argument specifies the buffer length in bytes. When passed, a comparison is made to ensure that the buffer size is at least as large as the size implied by the header. If this function fails for any reason, an array::header::exception is thrown.

Common Methods

The following methods are common to 1D, 2D, and 3D arrays, but are implemented in the array class specific to each dimensionality rather than in the base class.

size_t array::size() const

Total number of elements in array, e.g., nx × ny × nz for 3D arrays.

size_t array::cache_size() const

Return the cache size in number of bytes.

void array::set_cache_size(size_t csize)

Set minimum cache size in bytes. The actual size is always a power of two bytes and consists of at least one block. If csize is zero, then a default cache size is used, which requires the array dimensions to be known.

void array::get(Scalar *p) const

Decompress entire array and store at p, for which sufficient storage must have been allocated. The uncompressed array is assumed to be contiguous (with default strides) and stored in the usual “row-major” order, i.e., with x varying faster than y and y varying faster than z.

void array::set(const Scalar *p)

Initialize array by copying and compressing data stored at p. The uncompressed data is assumed to be stored as in the get() method.

Scalar array::operator[](uint index) const

Return scalar stored at given flat index (inspector). For a 3D array, index = x + nx * (y + ny * z).

reference array::operator[](uint index)

Return proxy reference to scalar stored at given flat index (mutator). For a 3D array, index = x + nx * (y + ny * z).

iterator array::begin()

Return iterator to beginning of array.

iterator array::end()

Return iterator to end of array. As with STL iterators, the end points to a virtual element just past the last valid array element.

1D, 2D, and 3D Arrays

Below are classes and methods specific to each array dimensionality and template scalar type (float or double). Since the classes and methods share obvious similarities regardless of dimensionality, only one generic description for all dimensionalities is provided.

Note: In the class declarations below, the class template for the scalar type is omitted for readability, e.g., class array1 is used as shorthand for template <typename Scalar> class array1. Wherever the type Scalar appears, it refers to this template argument.

class array1 : public array
class array2 : public array
class array3 : public array

This is a 1D/2D/3D array that inherits basic functionality from the generic array base class. The template argument, Scalar, specifies the floating type returned for array elements. The suffixes f and d can also be appended to each class to indicate float or double type, e.g., array1f is a synonym for array1<float>.

class arrayANY : public array

Fictitious class used to refer to any one of array1, array2, and array3. This class is not part of the zfp API.

array1::array1()
array2::array2()
array3::array3()

Default constructor. Creates an empty array whose size and rate are both zero.

Note

The default constructor is useful when the array size or rate is not known at time of construction. Before the array can become usable, however, it must be resized and its rate must be set via array::set_rate(). These two tasks can be performed in either order. Furthermore, the desired cache size should be set using array::set_cache_size(), as the default constructor creates a cache that holds only one zfp block, i.e., the minimum possible.

array1::array1(uint n, double rate, const Scalar *p = 0, size_t csize = 0)
array2::array2(uint nx, uint ny, double rate, const Scalar *p = 0, size_t csize = 0)
array3::array3(uint nx, uint ny, uint nz, double rate, const Scalar *p = 0, size_t csize = 0)

Constructor of array with dimensions n (1D), nx × ny (2D), or nx × ny × nz (3D) using rate bits per value, at least csize bytes of cache, and optionally initialized from flat, uncompressed array p. If csize is zero, a default cache size is chosen.

array1::array1(const array::header &h, const uchar *buffer = 0, size_t buffer_size_bytes = 0)
array2::array2(const array::header &h, const uchar *buffer = 0, size_t buffer_size_bytes = 0)
array3::array3(const array::header &h, const uchar *buffer = 0, size_t buffer_size_bytes = 0)

Constructor from previously serialized compressed array. Struct array::header contains array metadata, while optional buffer points to the compressed data that is to be copied to the array. The optional buffer_size_bytes argument specifies the buffer length. If the constructor fails, an array::header::exception is thrown. See array::construct() for further details on the buffer and buffer_size_bytes arguments.

array1::array1(const array1 &a)
array2::array2(const array2 &a)
array3::array3(const array3 &a)

Copy constructor. Performs a deep copy.

virtual array1::~array1()
virtual array2::~array2()
virtual array3::~array3()

Virtual destructor (allows for inheriting from zfp arrays).

array1 &array1::operator=(const array1 &a)
array2 &array2::operator=(const array2 &a)
array3 &array3::operator=(const array3 &a)

Assignment operator. Performs a deep copy.

uint array2::size_x() const
uint array2::size_y() const
uint array3::size_x() const
uint array3::size_y() const
uint array3::size_z() const

Return array dimensions.

void array1::resize(uint n, bool clear = true)
void array2::resize(uint nx, uint ny, bool clear = true)
void array3::resize(uint nx, uint ny, uint nz, bool clear = true)

Resize the array (all previously stored data will be lost). If clear is true, then the array elements are all initialized to zero.

Note

It is often desirable (though not a requirement) to also set the cache size when resizing an array, e.g., in proportion to the array size; see array::set_cache_size(). This is particularly important when the array is default constructed, which initializes the cache size to the minimum possible of only one zfp block.

Scalar array1::operator()(uint i) const
Scalar array2::operator()(uint i, uint j) const
Scalar array3::operator()(uint i, uint j, uint k) const

Return scalar stored at multi-dimensional index given by i, j, and k (inspector).

reference array1::operator()(uint i)
reference array2::operator()(uint i, uint j)
reference array3::operator()(uint i, uint j, uint k)

Return proxy reference to scalar stored at multi-dimensional index given by i, j, and k (mutator).

Caching

As mentioned above, the array classes maintain a software write-back cache of at least one uncompressed block. When a block in this cache is evicted (e.g., due to a conflict), it is compressed back to permanent storage only if it was modified while stored in the cache.

The size cache to use is specified by the user and is an important parameter that needs careful consideration in order to balance the extra memory usage, performance, and quality (recall that data loss is incurred only when a block is evicted from the cache and compressed). Although the best choice varies from one application to another, we suggest allocating at least two “layers” of blocks, e.g., 2 × (nx / 4) × (ny / 4) blocks for 3D arrays, for applications that stream through the array and perform stencil computations such as gathering data from neighboring elements. This allows limiting the cache misses to compulsory ones. If the csize parameter provided to the constructor is set to zero bytes, then a default cache size of at least √n blocks is used, where n is the total number of blocks contained in the array.

The cache size can be set during construction, or can be set at a later time via array::set_cache_size(). Note that if csize = 0, then the array dimensions must have already been specified for the default size to be computed correctly. When the cache is resized, it is first flushed if not already empty. The cache can also be flushed explicitly if desired by calling array::flush_cache(). To empty the cache without compressing any cached data, call array::clear_cache(). To query the byte size of the cache, use array::cache_size().

By default, a direct-mapped cache is used with a hash function that maps block indices to cache lines. A faster but more collision prone hash can be enabled by defining the preprocessor macro ZFP_WITH_CACHE_FAST_HASH. A two-way skew-associative cache is enabled by defining the preprocessor macro ZFP_WITH_CACHE_TWOWAY.

Serialization

zfp’s compressed arrays can be serialized to sequential, contiguous storage and later recovered back into an object, for example, to support I/O of compressed-array objects. Two pieces of information are needed to describe a zfp array: the raw compressed data, obtained via array::compressed_data() and array::compressed_size(), and a header that describes the array scalar type, dimensions, and rate, obtained via array::get_header(). The user may concatenate the header and compressed data to form a fixed-rate byte stream that can be read by the zfp command-line tool. When serializing the array, the user should first call array::flush_cache() before accessing the raw compressed data.

There are two primary ways to construct a compressed-array object from compressed data: via array-specific constructors and via a generic factory function:

  • When the array scalar type (i.e., float or double) and dimensionality (i.e., 1D, 2D, or 3D) are already known, the corresponding array constructor may be used. If the scalar type and dimensionality stored in the header do not match the array class, then an exception is thrown.

  • zfp provides a factory function that can be used when the serialized array type is unknown but described in the header. This function returns a pointer to the abstract base class, array, which the caller should dynamically cast to the corresponding derived array, e.g., by examining array::scalar_type() and array::dimensionality().

    The (static) factory function is made available by including zfpfactory.h. This header must be included after first including the header files associated with the compressed arrays, i.e., zfparray1.h, zfparray2.h, and zfparray3.h. Only those arrays whose header files are included can be constructed by the factory function. This design decouples the array classes so that they may be included independently, for example, to reduce compilation time.

Both types of deserialization functions accept an array::header, an optional buffer holding compressed data, and an optional buffer size. If this buffer is provided, then a separate copy of the compressed data it holds is made, which is used to initialize the array. If the optional buffer size is also provided, then these functions throw an exception if the size is not at least as large as is expected from the metadata stored in the header. This safeguard is implemented to avoid accessing memory beyond the end of the buffer. If no buffer is provided, then all array elements are default initialized to zero. The array may later be initialized by directly reading/copying data into the space pointed to by array::compressed_data().

Below is a simple example of serialization of a 3D compressed array of doubles (error checking has been omitted for clarity):

zfp::array3d a(nx, ny, nz, rate);
...
a.flush_cache();
zfp::array::header h = a.get_header();
fwrite(&h, sizeof(h), 1, file);
fwrite(a.compressed_data(), a.compressed_size(), 1, file);

We may then deserialize this array using the factory function. The following example reads the compressed data directly into the array without making a copy:

zfp::array::header h;
fread(&h, sizeof(h), 1, file);
zfp::array* p = zfp::array::construct(h);
fread(p->compressed_data(), p->compressed_size(), 1, file);
assert(p->dimensionality() == 3 && p->scalar_type() == zfp_type_double);
zfp::array3d& a = *dynamic_cast<zfp::array3d*>(p);

References

class array1::reference
class array2::reference
class array3::reference

Array indexing operators must return lvalue references that alias array elements and serve as vehicles for assigning values to those elements. Unfortunately, zfp cannot simply return a standard C++ reference (e.g., float&) to an uncompressed array element since the element in question may exist only in compressed form or as a transient cached entry that may be invalidated (evicted) at any point.

To address this, zfp provides proxies for references and pointers that act much like regular references and pointers, but which refer to elements by array and index rather than by memory address. When assigning to an array element through such a proxy reference or pointer, the corresponding element is decompressed to cache (if not already cached) and immediately updated.

zfp references may be freely passed to other functions and they remain valid during the lifetime of the corresponding array element. One may also take the address of a reference, which yields a proxy pointer. When a reference appears as an rvalue in an expression, it is implicitly converted to a value.

The following operators are defined for zfp references. They act on the referenced array element in the same manner as operators defined for conventional C++ references.

reference reference::operator=(const reference &ref)

Assignment (copy) operator. The referenced element, elem, is assigned the value stored at the element referenced by ref. Return *this.

reference reference::operator=(Scalar val)
reference reference::operator+=(Scalar val)
reference reference::operator-=(Scalar val)
reference reference::operator*=(Scalar val)
reference reference::operator/=(Scalar val)

Assignment and compound assignment operators. For a given operator op, update the referenced element, elem, via elem op val. Return *this.

pointer reference::operator&()

Return pointer to the referenced array element.

Finally, zfp proxy references serve as a building block for implementing proxy pointers and iterators.

Pointers

class array1::pointer
class array2::pointer
class array3::pointer

Similar to references, zfp supports proxies for pointers to individual array elements. From the user’s perspective, such pointers behave much like regular pointers to uncompressed data, e.g., instead of

float a[ny][nx];     // uncompressed 2D array of floats
float* p = &a[0][0]; // point to first array element
p[nx] = 1;           // set a[1][0] = 1
*++p = 2;            // set a[0][1] = 2

one would write

zfp::array2<float> a(nx, ny, rate);       // compressed 2D array of floats
zfp::array2<float>::pointer p = &a(0, 0); // point to first array element
p[nx] = 1;                                // set a(0, 1) = 1
*++p = 2;                                 // set a(1, 0) = 2

However, even though zfp’s proxy pointers point to individual scalars, they are associated with the array that those scalars are stored in, including the array’s dimensionality. Pointers into arrays of different dimensionality have incompatible type. Moreover, pointers to elements in different arrays are incompatible. For example, one cannot take the difference between pointers into two different arrays.

Unlike zfp’s proxy references, its proxy pointers support traversing arrays using conventional pointer arithmetic. In particular, unlike the iterators below, zfp’s pointers are oblivious to the fact that the compressed arrays are partitioned into blocks, and the pointers traverse arrays element by element as though the arrays were flattened in standard C row-major order. That is, if p points to the first element of a 3D array a(nx, ny, nz), then a(i, j, k) == p[i + nx * (j + ny * k)]. In other words, pointer indexing follows the same order as flat array indexing (see array::operator[]()).

A pointer remains valid during the lifetime of the scalar that it points to. Like conventional pointers, proxy pointers can be passed to other functions and manipulated there, for instance by passing the pointer by reference via pointer&.

The following operators are defined for proxy pointers. Below p refers to the pointer being acted upon.

pointer pointer::operator=(const pointer &q)

Assignment operator. Assigns q to p.

reference pointer::operator*() const

Dereference operator. Return proxy reference to the value pointed to by p.

reference pointer::operator[](ptrdiff_t d) const

Index operator. Return reference to the value stored at p[d].

pointer &pointer::operator++()
pointer &pointer::operator--()

Pre increment (decrement) pointer, e.g., ++p. Return reference to the incremented (decremented) pointer.

pointer pointer::operator++(int)
pointer pointer::operator--(int)

Post increment (decrement) pointer, e.g., p++. Return a copy of the pointer before it was incremented (decremented).

pointer pointer::operator+=(ptrdiff_t d)
pointer pointer::operator-=(ptrdiff_t d)

Increment (decrement) pointer by d. Return a copy of the incremented (decremented) pointer.

pointer pointer::operator+(ptrdiff_t d) const
pointer pointer::operator-(ptrdiff_t d) const

Return a copy of the pointer incremented (decremented) by d.

ptrdiff_t pointer::operator-(const pointer &q) const

Return difference p - q. Defined only for pointers within the same array.

bool pointer::operator==(const pointer &q) const
bool pointer::operator!=(const pointer &q) const

Pointer comparison. Return true (false) if p and q do (do not) point to the same array element.

Iterators

class array1::iterator
class array2::iterator
class array3::iterator

Iterators provide a mechanism for sequentially traversing a possibly multi-dimensional array without having to track array indices or bounds. They are also the preferred mechanism, compared to nested index loops, for initializing arrays, because they are guaranteed to visit the array one block at a time. This allows all elements of a block to be initialized together and ensures that the block is not compressed to memory before it has been fully initialized, which might otherwise result in poor compression and, consequently, larger errors than when the entire block is initialized as a whole. Note that the iterator traversal order differs in this respect from traversal by pointers.

The order of blocks visited is row-major (as in C), and the elements within a block are also visited in row-major order, i.e., first by x, then by y, and finally by z. All 4d values in a block are visited before moving on to the next block.

The iterators provided by zfp are sequential forward iterators, except for 1D array iterators, which are random access iterators. The reason why higher dimensional iterators do not support random access is that this would require very complicated index computations, especially for arrays with partial blocks. zfp iterators are STL compliant and can be used in STL algorithms that support forward and random access iterators.

All Iterators

Per STL mandate, the iterators define several types:

type iterator::value_type

The scalar type associated with the array that the iterator points into.

type iterator::difference_type

Difference between two iterators in number of array elements.

type iterator::reference
type iterator::pointer

The reference and pointer type associated with the iterator’s parent array class.

type iterator::iterator_category

Type of iterator: std::random_access_iterator_tag for 1D arrays; std::forward_iterator_tag for all other arrays.

The following operations are defined on iterators:

iterator iterator::operator=(const iterator &it)

Assignment (copy) operator. Make the iterator point to the same element as it.

reference iterator::operator*() const

Dereference operator. Return reference to the value pointed to by the iterator.

iterator &iterator::operator++()

Pre increment. Return a reference to the incremented iterator.

iterator iterator::operator++(int)

Post increment. Return the value of the iterator before being incremented.

bool iterator::operator==(const iterator &it) const
bool iterator::operator!=(const iterator &it) const

Return true (false) if the two iterators do (do not) point to the same element.

uint iterator::i() const
uint iterator::j() const
uint iterator::k() const

Return array index of element pointed to by the iterator. iterator::i() is defined for all arrays. iterator::j() is defined only for 2D and 3D arrays. iterator::k() is defined only for 3D arrays.

1D Array Iterators

The following operators are defined only for 1D arrays:

array1::reference array1::iterator::operator[](difference_type d) const

Random access index operator.

array1::iterator &array1::iterator::operator--()

Pre decrement. Return a reference to the decremented iterator.

array1::iterator array1::iterator::operator--(int)

Post decrement. Return the value of the iterator before being decremented.

array1::iterator array1::iterator::operator+=(difference_type d)
array1::iterator array1::iterator::operator-=(difference_type d)

Increment (decrement) iterator d times. Return value of incremented (decremented) iterator.

array1::iterator array1::iterator::operator+(difference_type d) const
array1::iterator array1::iterator::operator-(difference_type d) const

Return a new iterator that has been incremented (decremented) by d.

difference_type array1::iterator::operator-(const array1::iterator &it) const

Return difference between this iterator and it in number of elements. The iterators must refer to elements in the same array.

bool array1::iterator::operator<=(const array1::iterator &it) const
bool array1::iterator::operator>=(const array1::iterator &it) const
bool array1::iterator::operator<(const array1::iterator &it) const
bool array1::iterator::operator>(const array1::iterator &it) const

Return true if the two iterators satisfy the given relationship. For two iterators, p and q, within the same array, p < q if and only if q can be reached by incrementing p one or more times.

Views

zfp 0.5.4 adds array views. Much like how references allow indirect access to single array elements, views provide indirect access to whole arrays, or more generally to rectangular subsets of arrays. A view of an array does not allocate any storage for the array elements. Rather, the view accesses shared storage managed by the underlying array. This allows for multiple entries into an array without the need for expensive deep copies. In a sense, views can be thought of as shallow copies of arrays.

When a view exposes a whole array array<type>, it provides similar functionality to a C++ reference array<type>& or pointer array<type>* to the array. However, views are more general in that they also allow restricting access to a user-specified subset of the array, and unlike pointers also provide for the same syntax when accessing the array, e.g., array_view(i, j) instead of (*array_ptr)(i, j).

zfp’s nested views further provide for multidimensional array access analogous to the C/C++ nested array syntax array[i][j]. Finally, zfp’s private views can be used to ensure thread-safe access to its compressed arrays.

Note

Like iterators and proxy references and pointers, a view is valid only during the lifetime of the array that it references. No reference counting is done to keep the array alive. It is up to the user to ensure that the referenced array object is valid when accessed through a view.

There are several types of views distinguished by these attributes:

  • Read-only vs. read-write access.
  • Shared vs. private access.
  • Flat vs. nested indexing.

Each of these attributes is discussed in detail below in these sections:

Immutable view

The most basic view is the immutable const_view, which supports read-only access to the array elements it references. This view serves primarily as a base class for more specialized views. Its constructors allow establishing access to a whole array or to a rectangular subset of an array. Note that like references, pointers, and iterators, views are types nested within the arrays that they reference.

class array1::const_view
class array2::const_view
class array3::const_view

Immutable view into 1D, 2D, and 3D array.

array1::const_view::const_view(array1 *array)
array2::const_view::const_view(array2 *array)
array3::const_view::const_view(array3 *array)

Constructor for read-only access to a whole array. As already mentioned, these views are valid only during the lifetime of the underlying array object.

array1::const_view::const_view(array1 *array, uint x, uint nx)
array2::const_view::const_view(array2 *array, uint x, uint y, uint nx, uint ny)
array3::const_view::const_view(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)

Constructors for read-only access to a rectangular subset of an array. The subset is specified by an offset, e.g., (x, y, z) for a 3D array, and dimensions, e.g., (nx, ny, nz) for a 3D array. The rectangle must fit within the surrounding array.

uint array1::const_view::global_x(uint i) const
uint array2::const_view::global_x(uint i) const
uint array2::const_view::global_y(uint j) const
uint array3::const_view::global_x(uint i) const
uint array3::const_view::global_y(uint j) const
uint array3::const_view::global_z(uint k) const

Return global array index associated with local view index. For instance, if a 1D view has been constructed with offset x, then global_x(i) returns x + i.

uint array1::const_view::size_x() const
uint array2::const_view::size_x() const
uint array2::const_view::size_y() const
uint array3::const_view::size_x() const
uint array3::const_view::size_y() const
uint array3::const_view::size_z() const

Return dimensions of view.

Scalar array1::const_view::operator()(uint i) const
Scalar array2::const_view::operator()(uint i, uint j) const
Scalar array3::const_view::operator()(uint i, uint j, uint k) const

Return scalar stored at multi-dimensional index given by x + i, y + j, and z + k, where x, y, and z specify the offset into the array.

Scalar array1::const_view::operator[](uint index) const

Alternative inspector for 1D arrays identical to array1::const_view::operator()().

There are a number of common methods inherited from a base class, preview, further up the class hierarchy.

double arrayANY::const_view::rate() const

Return rate in bits per value. Same as array::rate().

size_t arrayANY::const_view::size() const

Total number of elements in view, e.g., nx × ny × nz for 3D views.

With the above definitions, the following example shows how a 2D view is constructed and accessed:

zfp::array2d a(200, 100, rate); // define 200x100 array of doubles
zfp::array2d::const_view v(&a, 10, 5, 20, 20); // v is a 20x20 view into array a
assert(v(2, 1) == a(12, 6)); // v(2, 1) == a(10 + 2, 5 + 1) == a(12, 6)
assert(v.size() == 400); // 20x20 == 400

Mutable view

The basic mutable view derives from the const_view but adds operators for write-access. Its constructors are similar to those for the const_view.

class array1::view : public array1::const_view
class array2::view : public array2::const_view
class array3::view : public array3::const_view

Mutable view into 1D, 2D, and 3D array.

array1::view::view(array1 *array)
array2::view::view(array2 *array)
array3::view::view(array3 *array)
array1::view::view(array1 *array, uint x, uint nx)
array2::view::view(array2 *array, uint x, uint y, uint nx, uint ny)
array3::view::view(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)

Whole-array and sub-array mutable view constructors. See const_view constructors for details.

array1::reference array1::view::operator()(uint i)
array2::reference array2::view::operator()(uint i, uint j)
array3::reference array3::view::operator()(uint i, uint j, uint k)

These operators, whose arguments have the same meaning as in the array accessors, return proxy references to individual array elements for write access.

Flat view

The views discussed so far require multidimensional indexing, e.g., (i, j, k) for 3D views. Some applications prefer one-dimensional linear indexing, which is provided by the specialized flat view. For example, in a 3D view with dimensions (nx, ny, nz), a multidimensional index (i, j, k) corresponds to the flat view index

index = i + nx * (j + ny * k)

This is true regardless of the view offset (x, y, z).

The flat view derives from the mutable view and adds operator[] for flat indexing. This operator is essentially equivalent to array::operator[]() defined for 2D and 3D arrays. Flat views also provide functions for converting between multidimensional and flat indices.

Flat views are available only for 2D and 3D arrays. The basic mutable view, array1::view, for 1D arrays can be thought of as either a flat or a nested view.

class array2::flat_view : public array2::view
class array3::flat_view : public array2::view

Flat, mutable views for 2D and 3D arrays.

array2::flat_view::flat_view(array2 *array)
array3::flat_view::flat_view(array3 *array)
array2::flat_view::flat_view(array2 *array, uint x, uint y, uint nx, uint ny)
array3::flat_view::flat_view(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)

Whole-array and sub-array flat view constructors. See const_view constructors for details.

uint array2::flat_view::index(uint i, uint j) const
uint array3::flat_view::index(uint i, uint j, uint k) const

Return flat index associated with multidimensional index.

void array2::flat_view::ij(uint &i, uint &j, uint index) const
void array3::flat_view::ijk(uint &i, uint &j, uint &k, uint index) const

Convert flat index to multidimensional index.

Scalar array2::flat_view::operator[](uint index) const
Scalar array3::flat_view::operator[](uint index) const

Return array element associated with given flat index.

array2::reference array2::flat_view::operator[](uint index)
array3::reference array3::flat_view::operator[](uint index)

Return reference to array element associated with given flat index.

Nested view

C and C++ support nested arrays (arrays of arrays), e.g., double a[10][20][30], which are usually accessed via nested indexing a[i][j][k]. Here a is a 3D array, a[i] is a 2D array, and a[i][j] is a 1D array. This 3D array can also be accessed via flat indexing, e.g.,

a[i][j][k] == (&a[0][0][0])[600 * i + 30 * j + k]

Nested views provide a mechanism to access array elements through nested indexing and to extract lower-dimensional “slices” of multidimensional arrays. Nested views are mutable.

Nested views are associated with a dimensionality. For instance, if v is a 3D nested view of a 3D array, then v[i] is a 2D nested view (of a 3D array), v[i][j] is a 1D nested view (of a 3D array), and v[i][j][k] is a scalar array element. Note that the order of indices is reversed when using nested indexing compared to multidimensional indexing, e.g., v(i, j, k) == v[k][j][i].

Whereas operator[] on an array object accesses an element through flat indexing, the same array can be accessed through a nested view to in effect provide nested array indexing:

zfp::array3d a(30, 20, 10, rate); // define 30x20x10 3D array
assert(a[32] == a(2, 1, 0)); // OK: flat and multidimensional indexing
assert(a[32] == a[0][1][2]); // ERROR: a does not support nested indexing
zfp::array3d::nested_view v(&a); // define a nested view of a
assert(a[32] == v[0][1][2]); // OK: v supports nested indexing
zfp::array2d b(v[5]); // define and deep copy 30x20 2D slice of a
assert(a(2, 1, 5) == b(2, 1)); // OK: multidimensional indexing
class array2::nested_view1

View of a 1D slice of a 2D array.

class array2::nested_view2

2D view of a 2D (sub)array.

class array3::nested_view1

View of a 1D slice of a 3D array.

class array3::nested_view2

View of a 2D slice of a 3D array.

class array3::nested_view3

3D view of a 3D (sub)array.

array2::nested_view2::nested_view2(array2 *array)
array3::nested_view3::nested_view3(array3 *array)
array2::nested_view2::nested_view2(array2 *array, uint x, uint y, uint nx, uint ny)
array3::nested_view3::nested_view3(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)

Whole-array and sub-array nested view constructors. See const_view constructors for details. Lower-dimensional view constructors are not accessible to the user but are invoked when accessing views via nested indexing.

array2::nested_view1::size_x() const
array2::nested_view2::size_x() const
array2::nested_view2::size_y() const
array3::nested_view1::size_x() const
array3::nested_view2::size_x() const
array3::nested_view2::size_y() const
array3::nested_view3::size_x() const
array3::nested_view3::size_y() const
array3::nested_view3::size_z() const

View dimensions.

array3::nested_view2 array3::nested_view3::operator[](uint index) const

Return view to a 2D slice of a 3D array.

array2::nested_view1 array2::nested_view2::operator[](uint index) const
array3::nested_view1 array3::nested_view2::operator[](uint index) const

Return view to a 1D slice of a 2D or 3D array.

Scalar array2::nested_view1::operator[](uint index) const
Scalar array3::nested_view1::operator[](uint index) const

Return scalar element of a 2D or 3D array.

array2::reference array2::nested_view1::operator[](uint index)
array3::reference array3::nested_view1::operator[](uint index)

Return reference to a scalar element of a 2D or 3D array.

Scalar array2::nested_view1::operator()(uint i) const
Scalar array2::nested_view2::operator()(uint i, uint j) const
Scalar array3::nested_view1::operator()(uint i) const
Scalar array3::nested_view2::operator()(uint i, uint j) const
Scalar array3::nested_view3::operator()(uint i, uint j, uint k) const

Return scalar element of a 2D or 3D array.

array2::reference array2::nested_view1::operator()(uint i)
array2::reference array2::nested_view2::operator()(uint i, uint j)
array3::reference array3::nested_view1::operator()(uint i)
array3::reference array3::nested_view2::operator()(uint i, uint j)
array3::reference array3::nested_view3::operator()(uint i, uint j, uint k)

Return reference to a scalar element of a 2D or 3D array.

Slicing

Arrays can be constructed as deep copies of slices of higher-dimensional arrays, as the code example above shows (i.e., zfp::array2d b(v[5]);). Unlike views, which have reference semantics, such array slicing has value semantics. In this example, 2D array b is initialized as a (deep) copy of a slice of 3D array a via nested view v. Subsequent modifications of b have no effect on a.

Slicing is implemented as array constructors templated on views. Upon initialization, elements are copied one at a time from the view via multidimensional indexing, e.g., v(i, j, k). Note that view and array dimensionalities must match, but aside from this an array may be constructed from any view.

Slicing needs not change the dimensionality, but can be used to copy an equidimensional subset of one array to another array, as in this example:

zfp::array3d a(30, 20, 10, rate);
zfp::array3d::const_view v(&a, 1, 2, 3, 4, 5, 6);
zfp::array3d b(v);
assert(b(0, 0, 0) == a(1, 2, 3));
assert(b.size_x() == 4);
assert(b.size_y() == 5);
assert(b.size_z() == 6);

Slicing adds the following templated array constructors.

template<class View>
array1::array1(const View &v)
template<class View>
array2::array2(const View &v)
template<class View>
array3::array3(const View &v)

Construct array from a view via a deep copy. The view, v, must support multidimensional indexing. The rate for the constructed array is initialized to the rate of the array associated with the view. Note that the actual rate may differ if the constructed array is a lower-dimensional slice of a higher-dimensional array due to lower rate granularity (see FAQ #12). The cache size of the constructed array is set to the default size.

Private immutable view

zfp’s compressed arrays are in general not thread-safe. The main reason for this is that each array maintains its own cache of uncompressed blocks. Race conditions on the cache would occur unless it were locked upon each and every array access, which would have a prohibitive performance cost.

To ensure thread-safe access, zfp provides private mutable and immutable views of arrays that maintain their own private caches. The private_const_view immutable view provides read-only access to the underlying array. It is similar to a const_view in this sense, but differs in that it maintains its own private cache rather than sharing the cache owned by the array. Multiple threads may thus access the same array in parallel through their own private views.

Note

Private views do not guarantee cache coherence. If, for example, the array is modified, then already cached data in a private view is not automatically updated. It is up to the user to ensure cache coherence by flushing (compressing modified blocks) or clearing (emptying) caches when appropriate.

The cache associated with a private view can be manipulated in the same way an array’s cache can. For instance, the user may set the cache size on a per-view basis.

Unlike with private mutable views, private immutable views may freely access any element in the array visible through the view, i.e., multiple threads may read the same array element simultaneously. For an example of how to use private views for both read and write multithreaded access, see the diffusion code example.

Private views support only multidimensional indexing, i.e., they are neither flat nor nested.

class array1::private_const_view
class array2::private_const_view
class array3::private_const_view

Immutable views of 1D, 2D, and 3D arrays with private caches.

array1::private_const_view::private_const_view(array1 *array)
array2::private_const_view::private_const_view(array2 *array)
array3::private_const_view::private_const_view(array3 *array)
array1::private_const_view::private_const_view(array1 *array, uint x, uint nx)
array2::private_const_view::private_const_view(array2 *array, uint x, uint y, uint nx, uint ny)
array3::private_const_view::private_const_view(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)

Whole-array and sub-array private immutable view constructors. See const_view constructors for details.

array1::private_const_view::size_x() const
array2::private_const_view::size_x() const
array2::private_const_view::size_y() const
array3::private_const_view::size_x() const
array3::private_const_view::size_y() const
array3::private_const_view::size_z() const

View dimensions.

Scalar array1::private_const_view::operator()(uint i) const
Scalar array2::private_const_view::operator()(uint i, uint j) const
Scalar array3::private_const_view::operator()(uint i, uint j, uint k) const

Return scalar element of a 1D, 2D, or 3D array.

The following functions are common among all dimensionalities:

size_t arrayANY::private_const_view::cache_size() const
void arrayANY::private_const_view::set_cache_size(size_t csize)
void arrayANY::private_const_view::clear_cache() const

Cache manipulation. See Caching for details.

Private mutable view

The mutable private_view supports both read and write access and is backed by a private cache. Because block compression, as needed to support write access, is not an atomic operation, mutable views and multithreading imply potential race conditions on the compressed blocks stored by an array. Although locking the array or individual blocks upon compression would be a potential solution, this would either serialize compression, thus hurting performance, or add a possibly large memory overhead by maintaining a lock with each block.

Note

To avoid multiple threads simultaneously compressing the same block, private mutable views of an array must reference disjoint, block-aligned subarrays for thread-safe access. Each block of 4d array elements must be associated with at most one private mutable view, and therefore these views must reference non-overlapping rectangular subsets that are aligned on block boundaries, except possibly for partial blocks on the array boundary. (Expert users may alternatively ensure serialization of block compression calls and cache coherence in other ways, in which case overlapping private views may be permitted.)

Aside from this requirement, the user may partition the array into disjoint views in whatever manner is suitable for the application. The private_view API supplies a very basic partitioner to facilitate this task, but may not result in optimal partitions or good load balance.

When multithreaded write access is desired, any direct accesses to the array itself (i.e., not through a view) could invoke compression. Even a read access may trigger compression if a modified block is evicted from the cache. Hence, such direct array accesses should be confined to serial code sections when private views are used.

As with private immutable views, cache coherence is not enforced. Although this is less of an issue for private mutable views due to the requirement that views may not overlap, each private mutable view overlaps an index space with the underlying array whose cache is not automatically synchronized with the view’s private cache. See the diffusion for an example of how to enforce cache coherence with mutable and immutable private views.

The private_view class inherits all public functions from private_const_view.

class array1::private_view : public array1::private_const_view
class array2::private_view : public array2::private_const_view
class array3::private_view : public array3::private_const_view

Mutable views of 1D, 2D, and 3D arrays with private caches.

class array1::private_view::view_reference
class array2::private_view::view_reference
class array3::private_view::view_reference

Proxy references to array elements specialized for mutable private views.

array1::private_view::private_view(array1 *array)
array2::private_view::private_view(array2 *array)
array3::private_view::private_view(array3 *array)
array1::private_view::private_view(array1 *array, uint x, uint nx)
array2::private_view::private_view(array2 *array, uint x, uint y, uint nx, uint ny)
array3::private_view::private_view(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)

Whole-array and sub-array private mutable view constructors. See const_view constructors for details.

array1::private_view::view_reference array1::private_view::operator()(uint i) const
array2::private_view::view_reference array2::private_view::operator()(uint i, uint j) const
array3::private_view::view_reference array3::private_view::operator()(uint i, uint j, uint k) const

Return reference to a scalar element of a 1D, 2D, or 3D array.

The following functions are common among all dimensionalities:

void arrayANY::private_view::partition(uint index, uint count)

Partition the current view into count roughly equal-size pieces along the view’s longest dimension and modify the view’s extents to match the piece indexed by index, with 0 ≤ index < count. These functions may be called multiple times, e.g., to recursively partition along different dimensions. The partitioner does not generate new views; it merely modifies the current values of the view’s offsets and dimensions. Note that this may result in empty views whose dimensions are zero, e.g., if there are more pieces than blocks along a dimension.

void arrayANY::private_view::flush_cache() const

Flush cache by compressing any modified blocks and emptying the cache.

C bindings

zfp 0.5.4 adds cfp: C language bindings for compressed arrays via wrappers around the C++ classes.

The C API has been designed to facilitate working with compressed arrays without the benefits of C++ operator overloading and self-aware objects, which greatly simplify the syntax. Whereas one possible design considered is to map each C++ method to a C function with a prefix, such as zfp_array3d_get(a, i, j, k) in place of a(i, j, k) for accessing an element of a 3D array of doubles, such code would quickly become unwieldy when part of longer expressions.

Instead, cfp uses the notion of nested C namespaces that are structs of function pointers, such as cfp.array3d. Although this may seem no more concise than a design based on prefixes, the user may alias these namespaces (somewhat similar to C++ using namespace declarations) using far shorter names via C macros or local variables. For instance:

const cfp_array3d_api _ = cfp.array3d; // _ is a namespace alias
cfp_array3d* a = _.ctor(nx, ny, nz, rate, 0, 0);
double value = _.get(a, i, j, k);
_.set(a, i, j, k, value + 1);

which is a substitute for the C++ code

zfp::array3d a(nx, ny, nz, rate, 0, 0);
double value = a(i, j, k);
a(i, j, k) = value + 1;

Because the underlying C++ objects have no corresponding C representation, and because C objects are not self aware (they have no implicit this pointer), the C interface interacts with compressed arrays through pointers to opaque types that cfp converts to pointers to the corresponding C++ objects. As a consequence, cfp compressed arrays must be allocated on the heap and explicitly freed via designated destructor functions to avoid memory leaks. The C++ constructors are mapped to C by allocating objects via C++ new. Moreover, the C API requires passing an array self pointer in order to manipulate the array.

As with the C++ classes, array elements can be accessed via multidimensional array indexing, e.g., get(array, i, j), and via flat, linear indexing, e.g., get_flat(array, i + nx * j).

cfp_array1f
cfp_array2f
cfp_array3f
cfp_array1d
cfp_array2d
cfp_array3d

Opaque types for 1D, 2D, and 3D compressed arrays of floats and doubles.

cfp_array1f* cfp_array1f.ctor(uint nx, double rate, const float* p, size_t csize)
cfp_array2d* cfp_array1d.ctor(uint nx, double rate, const double* p, size_t csize)
cfp_array2f* cfp_array2f.ctor(uint nx, uint ny, double rate, const float* p, size_t csize)
cfp_array2d* cfp_array2d.ctor(uint nx, uint ny, double rate, const double* p, size_t csize)
cfp_array3f* cfp_array3f.ctor(uint nx, uint ny, uint nz, double rate, const float* p, size_t csize)
cfp_array3d* cfp_array3d.ctor(uint nx, uint ny, uint nz, double rate, const double* p, size_t csize)

Array constructors. If p is not NULL, then the array is initialized from uncompressed storage; otherwise the array is zero initialized. csize is the minimum size cache (in bytes) to use. If csize is zero, a default size is chosen.

float cfp_array1f.get(const cfp_array1f* a, uint i)
float cfp_array2f.get(const cfp_array2f* a, uint i, uint j)
float cfp_array3f.get(const cfp_array3f* a, uint i, uint j, uint k)
double cfp_array1d.get(const cfp_array1d* a, uint i)
double cfp_array2d.get(const cfp_array2d* a, uint i, uint j)
double cfp_array3d.get(const cfp_array3d* a, uint i, uint j, uint k)

Array accessors via multidimensional indexing.

void cfp_array1f.set(const cfp_array1f* a, uint i, float val)
void cfp_array2f.set(const cfp_array2f* a, uint i, uint j, float val)
void cfp_array3f.set(const cfp_array3f* a, uint i, uint j, uint k, float val)
void cfp_array1d.set(const cfp_array1d* a, uint i, double val)
void cfp_array2d.set(const cfp_array2d* a, uint i, uint j, double val)
void cfp_array3d.set(const cfp_array3d* a, uint i, uint j, uint k, double val)

Array mutators for assigning values to array elements via multidimensional indexing.

void cfp_array1f.get_array(const cfp_array1f* self, float* p)
void cfp_array1d.get_array(const cfp_array1d* self, double* p)
void cfp_array2f.get_array(const cfp_array2f* self, float* p)
void cfp_array2d.get_array(const cfp_array2d* self, double* p)
void cfp_array3f.get_array(const cfp_array3f* self, float* p)
void cfp_array3d.get_array(const cfp_array3d* self, double* p)

Decompress entire array; see array::get().

void cfp_array1f.set_array(cfp_array1f* self, const float* p)
void cfp_array1d.set_array(cfp_array1d* self, const double* p)
void cfp_array2f.set_array(cfp_array2f* self, const float* p)
void cfp_array2d.set_array(cfp_array2d* self, const double* p)
void cfp_array3f.set_array(cfp_array3f* self, const float* p)
void cfp_array3d.set_array(cfp_array3d* self, const double* p)

Initialize entire array; see array::set().

uint cfp_array2f.size_x(const cfp_array2f* self)
uint cfp_array2f.size_y(const cfp_array2f* self)
uint cfp_array3f.size_x(const cfp_array3f* self)
uint cfp_array3f.size_y(const cfp_array3f* self)
uint cfp_array3f.size_z(const cfp_array3f* self)
uint cfp_array2d.size_x(const cfp_array2d* self)
uint cfp_array2d.size_y(const cfp_array2d* self)
uint cfp_array3d.size_x(const cfp_array3d* self)
uint cfp_array3d.size_y(const cfp_array3d* self)
uint cfp_array3d.size_z(const cfp_array3d* self)

Array dimensions.

void cfp_array1f.resize(cfp_array1f* self, uint n, int clear)
void cfp_array2f.resize(cfp_array2f* self, uint nx, uint ny, int clear)
void cfp_array3f.resize(cfp_array3f* self, uint nx, uint ny, uint nz, int clear)
void cfp_array1d.resize(cfp_array1d* self, uint n, int clear)
void cfp_array2d.resize(cfp_array2d* self, uint nx, uint ny, int clear)
void cfp_array3d.resize(cfp_array3d* self, uint nx, uint ny, uint nz, int clear)

Resize array.

The six array types share many functions that have the same signature. Below, each instance of cfp_array generically refers to one of those six types.

cfp_array* cfp_array.ctor_default()

Default constructor. Allocate an empty array that later can be resized and whose rate and cache size can be set by cfp_array.set_rate() and cfp_array.set_cache_size(). Return a pointer to the constructed array.

cfp_array* cfp_array.ctor_copy(const cfp_array* src)

Copy constructor. Return a pointer to the constructed array.

void cfp_array.dtor(cfp_array* self)

Destructor. The destructor not only deallocates any compressed data owned by the array, but also frees memory for itself, invalidating the self pointer upon return. Note that the user must explicitly call the destructor to avoid memory leaks.

void cfp_array.deep_copy(cfp_array* self, const cfp_array* src)

Perform a deep copy of src analogous to the C++ assignment operator.

double cfp_array.rate(const cfp_array* self)

See array::rate().

double cfp_array.set_rate(cfp_array* self, double rate)

See array::set_rate().

size_t cfp_array.cache_size(const cfp_array* self)

See array::cache_size().

void cfp_array.set_cache_size(cfp_array* self, size_t csize)

See array::set_cache_size().

void cfp_array.clear_cache(const cfp_array* self)

See array::clear_cache().

void cfp_array.flush_cache(const cfp_array* self)

See array::flush_cache().

size_t cfp_array.compressed_size(const cfp_array* self)

See array::compressed_size().

uchar* cfp_array.compressed_data(const cfp_array* self)

See array::compressed_data().

size_t cfp_array.size(const cfp_array* self)

See array::size().

float cfp_array.get_flat(const cfp_array* a, uint index)
double cfp_array.get_flat(const cfp_array* a, uint index)

Flat index array accessors; see array::operator[]().

void cfp_array.set_flat(const cfp_array* a, uint index, float val)
void cfp_array.set_flat(const cfp_array* a, uint index, double val)

Flat index array mutators; set array element with flat index to val.