Compressed Arrays¶
zfp’s compressed arrays are C++ classes, plus C wrappers around these classes, that implement random-accessible single- and multi-dimensional floating-point arrays whose storage size, specified in number of bits per array element, is set by the user. Such arbitrary storage is achieved via zfp’s lossy fixed-rate compression mode, by partitioning each d-dimensional array into blocks of 4d values and compressing each block to a fixed number of bits. The more smoothly the array values vary along each dimension, the more accurately zfp can represent them. In other words, these arrays are not suitable for representing data where adjacent elements are not correlated. Rather, the expectation is that the array represents a regularly sampled and predominantly continuous function, such as a temperature field in a physics simulation.
The rate, measured in number of bits per array element, can be specified in fractions of a bit (but see FAQs #12 and #18 for limitations). Note that array dimensions need not be multiples of four; zfp transparently handles partial blocks on array boundaries.
The C++ templated array classes are implemented entirely as header files that call the zfp C library to perform compression and decompression. These arrays cache decompressed blocks to reduce the number of compression and decompression calls. Whenever an array value is read, the corresponding block is first looked up in the cache, and if found the uncompressed value is returned. Otherwise the block is first decompressed and stored in the cache. Whenever an array element is written (whether actually modified or not), a “dirty bit” is set with its cached block to indicate that the block must be compressed back to persistent storage when evicted from the cache.
This section documents the public interface to the array classes, including base classes and member accessor classes like proxy references/pointers, iterators, and views.
The following sections are available:
Array Classes¶
Currently there are six array classes for 1D, 2D, and 3D arrays, each of which can represent single- or double-precision values. Although these arrays store values in a form different from conventional single- and double-precision floating point, the user interacts with the arrays via floats and doubles.
The array classes can often serve as direct substitutes for C/C++
single- and multi-dimensional floating-point arrays and STL vectors, but
have the benefit of allowing fine control over storage size. All classes
below belong to the zfp
namespace.
Base Class¶
-
class
array
¶ Virtual base class for common array functionality.
-
double
array::
rate
() const¶ Return rate in bits per value.
-
double
array::
set_rate
(double rate)¶ Set desired compression rate in bits per value. Return the closest rate supported. See FAQ #12 and FAQ #18 for discussions of the rate granularity. This method destroys the previous contents of the array.
-
virtual void
array::
clear_cache
() const¶ Empty cache without compressing modified cached blocks, i.e., discard any cached updates to the array.
-
virtual void
array::
flush_cache
() const¶ Flush cache by compressing all modified cached blocks back to persistent storage and emptying the cache. This method should be called before writing the compressed representation of the array to disk, for instance.
-
size_t
array::
compressed_size
() const¶ Return number of bytes of storage for the compressed data. This amount does not include the small overhead of other class members or the size of the cache. Rather, it reflects the size of the memory buffer returned by
compressed_data()
.
-
uchar *
array::
compressed_data
() const¶ Return pointer to compressed data for read or write access. The size of the buffer is given by
compressed_size()
.
-
uint
array::
dimensionality
() const¶ Return the dimensionality (1, 2, or 3) of the array.
-
array::header
array::
get_header
() const¶ Return a short fixed-length header describing the scalar type, dimensions, and rate associated with the array. An
array::header::exception
is thrown if the header cannot describe the array.
-
static array *
array::
construct
(const array::header &h, const uchar *buffer = 0, size_t buffer_size_bytes = 0)¶ Construct a compressed-array object whose scalar type, dimensions, and rate are given by the header h. Return a pointer to the base class upon success. The optional buffer points to compressed data that, when passed, is copied into the array. If buffer is absent, the array is default initialized with all zeroes. The optional buffer_size_bytes argument specifies the buffer length in bytes. When passed, a comparison is made to ensure that the buffer size is at least as large as the size implied by the header. If this function fails for any reason, an
array::header::exception
is thrown.
Common Methods¶
The following methods are common to 1D, 2D, and 3D arrays, but are implemented in the array class specific to each dimensionality rather than in the base class.
-
size_t
array::
size
() const¶ Total number of elements in array, e.g., nx × ny × nz for 3D arrays.
-
size_t
array::
cache_size
() const¶ Return the cache size in number of bytes.
-
void
array::
set_cache_size
(size_t csize)¶ Set minimum cache size in bytes. The actual size is always a power of two bytes and consists of at least one block. If csize is zero, then a default cache size is used, which requires the array dimensions to be known.
-
void
array::
get
(Scalar *p) const¶ Decompress entire array and store at p, for which sufficient storage must have been allocated. The uncompressed array is assumed to be contiguous (with default strides) and stored in the usual “row-major” order, i.e., with x varying faster than y and y varying faster than z.
-
void
array::
set
(const Scalar *p)¶ Initialize array by copying and compressing data stored at p. The uncompressed data is assumed to be stored as in the
get()
method.
-
Scalar
array::
operator[]
(uint index) const¶ Return scalar stored at given flat index (inspector). For a 3D array,
index = x + nx * (y + ny * z)
.
-
reference
array::
operator[]
(uint index)¶ Return proxy reference to scalar stored at given flat index (mutator). For a 3D array,
index = x + nx * (y + ny * z)
.
-
iterator
array::
begin
()¶ Return iterator to beginning of array.
-
iterator
array::
end
()¶ Return iterator to end of array. As with STL iterators, the end points to a virtual element just past the last valid array element.
1D, 2D, and 3D Arrays¶
Below are classes and methods specific to each array dimensionality and
template scalar type (float
or double
). Since the classes
and methods share obvious similarities regardless of dimensionality, only
one generic description for all dimensionalities is provided.
Note: In the class declarations below, the class template for the scalar
type is omitted for readability, e.g.,
class array1
is used as shorthand for
template <typename Scalar> class array1
. Wherever the type
Scalar
appears, it refers to this template argument.
-
class
array3
: public array¶ This is a 1D/2D/3D array that inherits basic functionality from the generic
array
base class. The template argument,Scalar
, specifies the floating type returned for array elements. The suffixesf
andd
can also be appended to each class to indicate float or double type, e.g.,array1f
is a synonym forarray1<float>
.
-
class
arrayANY
: public array¶ Fictitious class used to refer to any one of
array1
,array2
, andarray3
. This class is not part of the zfp API.
-
array1::
array1
()¶
-
array2::
array2
()¶
-
array3::
array3
()¶ Default constructor. Creates an empty array whose size and rate are both zero.
Note
The default constructor is useful when the array size or rate is not known at
time of construction. Before the array can become usable, however, it must
be resized and its rate must be set via
array::set_rate()
. These two tasks can be performed in either order.
Furthermore, the desired cache size should be set using
array::set_cache_size()
, as the default constructor creates a
cache that holds only one zfp block, i.e., the minimum possible.
-
array1::
array1
(uint n, double rate, const Scalar *p = 0, size_t csize = 0)¶
-
array2::
array2
(uint nx, uint ny, double rate, const Scalar *p = 0, size_t csize = 0)¶
-
array3::
array3
(uint nx, uint ny, uint nz, double rate, const Scalar *p = 0, size_t csize = 0)¶ Constructor of array with dimensions n (1D), nx × ny (2D), or nx × ny × nz (3D) using rate bits per value, at least csize bytes of cache, and optionally initialized from flat, uncompressed array p. If csize is zero, a default cache size is chosen.
-
array3::
array3
(const array::header &h, const uchar *buffer = 0, size_t buffer_size_bytes = 0)¶ Constructor from previously serialized compressed array. Struct
array::header
contains array metadata, while optional buffer points to the compressed data that is to be copied to the array. The optional buffer_size_bytes argument specifies the buffer length. If the constructor fails, anarray::header::exception
is thrown. Seearray::construct()
for further details on the buffer and buffer_size_bytes arguments.
-
virtual
array1::
~array1
()¶
-
virtual
array2::
~array2
()¶
-
virtual
array3::
~array3
()¶ Virtual destructor (allows for inheriting from zfp arrays).
-
uint
array2::
size_x
() const¶
-
uint
array2::
size_y
() const¶
-
uint
array3::
size_x
() const¶
-
uint
array3::
size_y
() const¶
-
uint
array3::
size_z
() const¶ Return array dimensions.
-
void
array1::
resize
(uint n, bool clear = true)¶
-
void
array2::
resize
(uint nx, uint ny, bool clear = true)¶
-
void
array3::
resize
(uint nx, uint ny, uint nz, bool clear = true)¶ Resize the array (all previously stored data will be lost). If clear is true, then the array elements are all initialized to zero.
Note
It is often desirable (though not a requirement) to also set the cache size
when resizing an array, e.g., in proportion to the array size;
see array::set_cache_size()
. This is particularly important when
the array is default constructed, which initializes the cache size to the
minimum possible of only one zfp block.
-
Scalar
array1::
operator()
(uint i) const¶
-
Scalar
array2::
operator()
(uint i, uint j) const¶
-
Scalar
array3::
operator()
(uint i, uint j, uint k) const¶ Return scalar stored at multi-dimensional index given by i, j, and k (inspector).
-
reference
array3::
operator()
(uint i, uint j, uint k)¶ Return proxy reference to scalar stored at multi-dimensional index given by i, j, and k (mutator).
Caching¶
As mentioned above, the array classes maintain a software write-back cache of at least one uncompressed block. When a block in this cache is evicted (e.g., due to a conflict), it is compressed back to permanent storage only if it was modified while stored in the cache.
The size cache to use is specified by the user and is an important parameter that needs careful consideration in order to balance the extra memory usage, performance, and quality (recall that data loss is incurred only when a block is evicted from the cache and compressed). Although the best choice varies from one application to another, we suggest allocating at least two “layers” of blocks, e.g., 2 × (nx / 4) × (ny / 4) blocks for 3D arrays, for applications that stream through the array and perform stencil computations such as gathering data from neighboring elements. This allows limiting the cache misses to compulsory ones. If the csize parameter provided to the constructor is set to zero bytes, then a default cache size of at least √n blocks is used, where n is the total number of blocks contained in the array.
The cache size can be set during construction, or can be set at a later
time via array::set_cache_size()
. Note that if csize = 0, then
the array dimensions must have already been specified for the default size
to be computed correctly. When the cache is resized, it is first flushed
if not already empty. The cache can also be flushed explicitly if desired
by calling array::flush_cache()
. To empty the cache without
compressing any cached data, call array::clear_cache()
. To query
the byte size of the cache, use array::cache_size()
.
By default, a direct-mapped cache is used with a hash function that maps
block indices to cache lines. A faster but more collision prone hash
can be enabled by defining the preprocessor macro
ZFP_WITH_CACHE_FAST_HASH
.
A two-way skew-associative cache is enabled by defining the preprocessor
macro ZFP_WITH_CACHE_TWOWAY
.
Serialization¶
zfp’s compressed arrays can be serialized to sequential, contiguous
storage and later recovered back into an object, for example, to support
I/O of compressed-array objects. Two pieces of information are needed
to describe a zfp array: the raw compressed data, obtained via
array::compressed_data()
and array::compressed_size()
,
and a header that describes the array scalar type,
dimensions, and rate, obtained via array::get_header()
.
The user may concatenate the header and compressed data to form a
fixed-rate byte stream that can be read by the zfp
command-line tool. When serializing the array,
the user should first call array::flush_cache()
before
accessing the raw compressed data.
There are two primary ways to construct a compressed-array object from compressed data: via array-specific constructors and via a generic factory function:
When the array scalar type (i.e.,
float
ordouble
) and dimensionality (i.e., 1D, 2D, or 3D) are already known, the corresponding array constructor may be used. If the scalar type and dimensionality stored in the header do not match the array class, then an exception is thrown.zfp provides a factory function that can be used when the serialized array type is unknown but described in the header. This function returns a pointer to the abstract base class,
array
, which the caller should dynamically cast to the corresponding derived array, e.g., by examiningarray::scalar_type()
andarray::dimensionality()
.The (static) factory function is made available by including
zfpfactory.h
. This header must be included after first including the header files associated with the compressed arrays, i.e.,zfparray1.h
,zfparray2.h
, andzfparray3.h
. Only those arrays whose header files are included can be constructed by the factory function. This design decouples the array classes so that they may be included independently, for example, to reduce compilation time.
Both types of deserialization functions accept an array::header
,
an optional buffer holding compressed data, and an optional buffer size.
If this buffer is provided, then a separate copy of the compressed data it
holds is made, which is used to initialize the array. If the optional buffer
size is also provided, then these functions throw an exception if the size
is not at least as large as is expected from the metadata stored in the
header. This safeguard is implemented to avoid accessing memory beyond the
end of the buffer. If no buffer is provided, then all array elements are
default initialized to zero. The array may later be initialized by directly
reading/copying data into the space pointed to by
array::compressed_data()
.
Below is a simple example of serialization of a 3D compressed array of doubles (error checking has been omitted for clarity):
zfp::array3d a(nx, ny, nz, rate);
...
a.flush_cache();
zfp::array::header h = a.get_header();
fwrite(&h, sizeof(h), 1, file);
fwrite(a.compressed_data(), a.compressed_size(), 1, file);
We may then deserialize this array using the factory function. The following example reads the compressed data directly into the array without making a copy:
zfp::array::header h;
fread(&h, sizeof(h), 1, file);
zfp::array* p = zfp::array::construct(h);
fread(p->compressed_data(), p->compressed_size(), 1, file);
assert(p->dimensionality() == 3 && p->scalar_type() == zfp_type_double);
zfp::array3d& a = *dynamic_cast<zfp::array3d*>(p);
Header¶
Short 12-byte headers are used to describe array metadata and compression
parameters when serializing a compressed array. This header is the same as
supported by the zfp_read_header()
and zfp_write_header()
functions, using ZFP_HEADER_FULL
to indicate that complete metadata
is to be stored in the header. The header is also compatible with the zfp
command-line tool. Processing of the header may result in an
exception being thrown.
-
class
array::
header
¶ The header stores information such as scalar type, array dimensions, and rate. Compressed-array headers are always 96 bits long. These bits are stored in the
header::buffer
field, the only field represented in the header.struct header { uchar buffer[BITS_TO_BYTES(ZFP_HEADER_SIZE_BITS)]; };
-
class
array::header::
exception
: public std::runtime_error¶ Compressed arrays can throw this exception upon serialization, when forming a header via
get_header()
, or deserialization, when constructing a compressed array via its constructor or factory function.
Note
Compressed-array headers use zfp’s most concise representation of only
96 bits. Such short headers support compressed blocks up to 2048 bits long.
This implies that the highest rate for 3D arrays is 2048/43 = 32
bits/value. 3D arrays whose rate exceeds 32 cannot be serialized using
array::get_header()
, which for such arrays throws an exception.
1D and 2D arrays do not suffer from this limitation.
References¶
-
class
array1::
reference
¶
-
class
array2::
reference
¶
-
class
array3::
reference
¶
Array indexing operators must return lvalue references that
alias array elements and serve as vehicles for assigning values to those
elements. Unfortunately, zfp cannot simply return a standard C++ reference
(e.g., float&
) to an uncompressed array element since the element in
question may exist only in compressed form or as a transient cached entry that
may be invalidated (evicted) at any point.
To address this, zfp provides proxies for references and pointers that act much like regular references and pointers, but which refer to elements by array and index rather than by memory address. When assigning to an array element through such a proxy reference or pointer, the corresponding element is decompressed to cache (if not already cached) and immediately updated.
zfp references may be freely passed to other functions and they remain valid during the lifetime of the corresponding array element. One may also take the address of a reference, which yields a proxy pointer. When a reference appears as an rvalue in an expression, it is implicitly converted to a value.
The following operators are defined for zfp references. They act on the referenced array element in the same manner as operators defined for conventional C++ references.
-
reference
reference::
operator=
(const reference &ref)¶ Assignment (copy) operator. The referenced element, elem, is assigned the value stored at the element referenced by ref. Return
*this
.
-
reference
reference::
operator=
(Scalar val)¶
-
reference
reference::
operator+=
(Scalar val)¶
-
reference
reference::
operator-=
(Scalar val)¶
-
reference
reference::
operator*=
(Scalar val)¶
-
reference
reference::
operator/=
(Scalar val)¶ Assignment and compound assignment operators. For a given operator
op
, update the referenced element, elem, via elemop
val. Return*this
.
-
pointer
reference::
operator&
()¶ Return pointer to the referenced array element.
Finally, zfp proxy references serve as a building block for implementing proxy pointers and iterators.
Pointers¶
-
class
array1::
pointer
¶
-
class
array2::
pointer
¶
-
class
array3::
pointer
¶
Similar to references, zfp supports proxies for pointers to individual array elements. From the user’s perspective, such pointers behave much like regular pointers to uncompressed data, e.g., instead of
float a[ny][nx]; // uncompressed 2D array of floats
float* p = &a[0][0]; // point to first array element
p[nx] = 1; // set a[1][0] = 1
*++p = 2; // set a[0][1] = 2
one would write
zfp::array2<float> a(nx, ny, rate); // compressed 2D array of floats
zfp::array2<float>::pointer p = &a(0, 0); // point to first array element
p[nx] = 1; // set a(0, 1) = 1
*++p = 2; // set a(1, 0) = 2
However, even though zfp’s proxy pointers point to individual scalars, they are associated with the array that those scalars are stored in, including the array’s dimensionality. Pointers into arrays of different dimensionality have incompatible type. Moreover, pointers to elements in different arrays are incompatible. For example, one cannot take the difference between pointers into two different arrays.
Unlike zfp’s proxy references, its proxy pointers support traversing
arrays using conventional pointer arithmetic. In particular, unlike the
iterators below, zfp’s pointers are oblivious to the
fact that the compressed arrays are partitioned into blocks, and the pointers
traverse arrays element by element as though the arrays were flattened in
standard C row-major order. That is, if p
points to the first
element of a 3D array a(nx, ny, nz)
, then
a(i, j, k) == p[i + nx * (j + ny * k)]
. In other words, pointer
indexing follows the same order as flat array indexing
(see array::operator[]()
).
A pointer remains valid during the lifetime of the scalar that it points to.
Like conventional pointers, proxy pointers can be passed to other functions
and manipulated there, for instance by passing the pointer by reference via
pointer&
.
The following operators are defined for proxy pointers. Below p refers to the pointer being acted upon.
-
pointer
pointer::
operator=
(const pointer &q)¶ Assignment operator. Assigns q to p.
-
reference
pointer::
operator*
() const¶ Dereference operator. Return proxy reference to the value pointed to by p.
-
reference
pointer::
operator[]
(ptrdiff_t d) const¶ Index operator. Return reference to the value stored at
p[d]
.
-
pointer &
pointer::
operator++
()¶
-
pointer &
pointer::
operator--
()¶ Pre increment (decrement) pointer, e.g.,
++p
. Return reference to the incremented (decremented) pointer.
-
pointer
pointer::
operator++
(int)¶
-
pointer
pointer::
operator--
(int)¶ Post increment (decrement) pointer, e.g.,
p++
. Return a copy of the pointer before it was incremented (decremented).
-
pointer
pointer::
operator+=
(ptrdiff_t d)¶
-
pointer
pointer::
operator-=
(ptrdiff_t d)¶ Increment (decrement) pointer by d. Return a copy of the incremented (decremented) pointer.
-
pointer
pointer::
operator+
(ptrdiff_t d) const¶
-
pointer
pointer::
operator-
(ptrdiff_t d) const¶ Return a copy of the pointer incremented (decremented) by d.
-
ptrdiff_t
pointer::
operator-
(const pointer &q) const¶ Return difference p - q. Defined only for pointers within the same array.
-
bool
pointer::
operator==
(const pointer &q) const¶
-
bool
pointer::
operator!=
(const pointer &q) const¶ Pointer comparison. Return true (false) if p and q do (do not) point to the same array element.
Iterators¶
-
class
array1::
iterator
¶
-
class
array2::
iterator
¶
-
class
array3::
iterator
¶
Iterators provide a mechanism for sequentially traversing a possibly multi-dimensional array without having to track array indices or bounds. They are also the preferred mechanism, compared to nested index loops, for initializing arrays, because they are guaranteed to visit the array one block at a time. This allows all elements of a block to be initialized together and ensures that the block is not compressed to memory before it has been fully initialized, which might otherwise result in poor compression and, consequently, larger errors than when the entire block is initialized as a whole. Note that the iterator traversal order differs in this respect from traversal by pointers.
The order of blocks visited is row-major (as in C), and the elements within a block are also visited in row-major order, i.e., first by x, then by y, and finally by z. All 4d values in a block are visited before moving on to the next block.
The iterators provided by zfp are sequential forward iterators, except for 1D array iterators, which are random access iterators. The reason why higher dimensional iterators do not support random access is that this would require very complicated index computations, especially for arrays with partial blocks. zfp iterators are STL compliant and can be used in STL algorithms that support forward and random access iterators.
All Iterators¶
Per STL mandate, the iterators define several types:
-
type
iterator::
value_type
¶ The scalar type associated with the array that the iterator points into.
-
type
iterator::
difference_type
¶ Difference between two iterators in number of array elements.
-
type
iterator::
reference
¶
-
type
iterator::
pointer
¶ The reference and pointer type associated with the iterator’s parent array class.
-
type
iterator::
iterator_category
¶ Type of iterator:
std::random_access_iterator_tag
for 1D arrays;std::forward_iterator_tag
for all other arrays.
The following operations are defined on iterators:
-
iterator
iterator::
operator=
(const iterator &it)¶ Assignment (copy) operator. Make the iterator point to the same element as it.
-
reference
iterator::
operator*
() const¶ Dereference operator. Return reference to the value pointed to by the iterator.
-
iterator &
iterator::
operator++
()¶ Pre increment. Return a reference to the incremented iterator.
-
iterator
iterator::
operator++
(int)¶ Post increment. Return the value of the iterator before being incremented.
-
bool
iterator::
operator==
(const iterator &it) const¶
-
bool
iterator::
operator!=
(const iterator &it) const¶ Return true (false) if the two iterators do (do not) point to the same element.
-
uint
iterator::
i
() const¶
-
uint
iterator::
j
() const¶
-
uint
iterator::
k
() const¶ Return array index of element pointed to by the iterator.
iterator::i()
is defined for all arrays.iterator::j()
is defined only for 2D and 3D arrays.iterator::k()
is defined only for 3D arrays.
1D Array Iterators¶
The following operators are defined only for 1D arrays:
-
array1::reference
array1::iterator::
operator[]
(difference_type d) const¶ Random access index operator.
-
array1::iterator &
array1::iterator::
operator--
()¶ Pre decrement. Return a reference to the decremented iterator.
-
array1::iterator
array1::iterator::
operator--
(int)¶ Post decrement. Return the value of the iterator before being decremented.
-
array1::iterator
array1::iterator::
operator-=
(difference_type d)¶ Increment (decrement) iterator d times. Return value of incremented (decremented) iterator.
-
array1::iterator
array1::iterator::
operator-
(difference_type d) const¶ Return a new iterator that has been incremented (decremented) by d.
Views¶
zfp 0.5.4 adds array views. Much like how references allow indirect access to single array elements, views provide indirect access to whole arrays, or more generally to rectangular subsets of arrays. A view of an array does not allocate any storage for the array elements. Rather, the view accesses shared storage managed by the underlying array. This allows for multiple entries into an array without the need for expensive deep copies. In a sense, views can be thought of as shallow copies of arrays.
When a view exposes a whole array array<type>
, it provides
similar functionality to a C++ reference array<type>&
or
pointer array<type>*
to the array. However, views are more
general in that they also allow restricting access to a user-specified
subset of the array, and unlike pointers also provide for the same
syntax when accessing the array, e.g., array_view(i, j)
instead
of (*array_ptr)(i, j)
.
zfp’s nested views further provide for multidimensional
array access analogous to the C/C++ nested array syntax array[i][j]
.
Finally, zfp’s private views can be used to ensure thread-safe access
to its compressed arrays.
Note
Like iterators and proxy references and pointers, a view is valid only during the lifetime of the array that it references. No reference counting is done to keep the array alive. It is up to the user to ensure that the referenced array object is valid when accessed through a view.
There are several types of views distinguished by these attributes:
- Read-only vs. read-write access.
- Shared vs. private access.
- Flat vs. nested indexing.
Each of these attributes is discussed in detail below in these sections:
- Immutable view
- Mutable view
- Flat view
- Nested view
- Slicing
- Private immutable view
- Private mutable view
Immutable view¶
The most basic view is the immutable const_view
, which
supports read-only access to the array elements it references.
This view serves primarily as a base class for more specialized
views. Its constructors allow establishing access to a whole
array or to a rectangular subset of an array. Note that like
references, pointers, and iterators, views are types nested within
the arrays that they reference.
-
class
array1::
const_view
¶
-
class
array2::
const_view
¶
-
class
array3::
const_view
¶ Immutable view into 1D, 2D, and 3D array.
-
array3::const_view::
const_view
(array3 *array)¶ Constructor for read-only access to a whole array. As already mentioned, these views are valid only during the lifetime of the underlying array object.
-
array3::const_view::
const_view
(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)¶ Constructors for read-only access to a rectangular subset of an array. The subset is specified by an offset, e.g., (x, y, z) for a 3D array, and dimensions, e.g., (nx, ny, nz) for a 3D array. The rectangle must fit within the surrounding array.
-
uint
array1::const_view::
global_x
(uint i) const¶
-
uint
array2::const_view::
global_x
(uint i) const¶
-
uint
array2::const_view::
global_y
(uint j) const¶
-
uint
array3::const_view::
global_x
(uint i) const¶
-
uint
array3::const_view::
global_y
(uint j) const¶
-
uint
array3::const_view::
global_z
(uint k) const¶ Return global array index associated with local view index. For instance, if a 1D view has been constructed with offset x, then
global_x(i)
returns x + i.
-
uint
array1::const_view::
size_x
() const¶
-
uint
array2::const_view::
size_x
() const¶
-
uint
array2::const_view::
size_y
() const¶
-
uint
array3::const_view::
size_x
() const¶
-
uint
array3::const_view::
size_y
() const¶
-
uint
array3::const_view::
size_z
() const¶ Return dimensions of view.
-
Scalar
array1::const_view::
operator()
(uint i) const¶
-
Scalar
array2::const_view::
operator()
(uint i, uint j) const¶
-
Scalar
array3::const_view::
operator()
(uint i, uint j, uint k) const¶ Return scalar stored at multi-dimensional index given by x + i, y + j, and z + k, where x, y, and z specify the offset into the array.
-
Scalar
array1::const_view::
operator[]
(uint index) const¶ Alternative inspector for 1D arrays identical to
array1::const_view::operator()()
.
There are a number of common methods inherited from a base class,
preview
, further up the class hierarchy.
-
double
arrayANY::const_view::
rate
() const¶ Return rate in bits per value. Same as
array::rate()
.
-
size_t
arrayANY::const_view::
size
() const¶ Total number of elements in view, e.g., nx × ny × nz for 3D views.
With the above definitions, the following example shows how a 2D view is constructed and accessed:
zfp::array2d a(200, 100, rate); // define 200x100 array of doubles
zfp::array2d::const_view v(&a, 10, 5, 20, 20); // v is a 20x20 view into array a
assert(v(2, 1) == a(12, 6)); // v(2, 1) == a(10 + 2, 5 + 1) == a(12, 6)
assert(v.size() == 400); // 20x20 == 400
Mutable view¶
The basic mutable view
derives from the const_view
but
adds operators for write-access. Its constructors are similar to those
for the const_view
.
-
class
array1::
view
: public array1::const_view¶
-
class
array2::
view
: public array2::const_view¶
-
class
array3::
view
: public array3::const_view¶ Mutable view into 1D, 2D, and 3D array.
-
array3::view::
view
(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)¶ Whole-array and sub-array mutable view constructors. See const_view constructors for details.
-
array3::reference
array3::view::
operator()
(uint i, uint j, uint k)¶ These operators, whose arguments have the same meaning as in the array accessors, return proxy references to individual array elements for write access.
Flat view¶
The views discussed so far require multidimensional indexing, e.g., (i, j, k) for 3D views. Some applications prefer one-dimensional linear indexing, which is provided by the specialized flat view. For example, in a 3D view with dimensions (nx, ny, nz), a multidimensional index (i, j, k) corresponds to the flat view index
index = i + nx * (j + ny * k)
This is true regardless of the view offset (x, y, z).
The flat view derives from the mutable view and adds operator[]
for flat indexing. This operator is essentially equivalent to
array::operator[]()
defined for 2D and 3D arrays. Flat views
also provide functions for converting between multidimensional and flat
indices.
Flat views are available only for 2D and 3D arrays. The basic mutable
view, array1::view
, for 1D arrays can be thought of as
either a flat or a nested view.
-
array3::flat_view::
flat_view
(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)¶ Whole-array and sub-array flat view constructors. See const_view constructors for details.
-
uint
array2::flat_view::
index
(uint i, uint j) const¶
-
uint
array3::flat_view::
index
(uint i, uint j, uint k) const¶ Return flat index associated with multidimensional index.
-
void
array2::flat_view::
ij
(uint &i, uint &j, uint index) const¶
-
void
array3::flat_view::
ijk
(uint &i, uint &j, uint &k, uint index) const¶ Convert flat index to multidimensional index.
-
Scalar
array2::flat_view::
operator[]
(uint index) const¶
-
Scalar
array3::flat_view::
operator[]
(uint index) const¶ Return array element associated with given flat index.
Nested view¶
C and C++ support nested arrays (arrays of arrays), e.g.,
double a[10][20][30]
, which are usually accessed via nested indexing
a[i][j][k]
. Here a
is a 3D array, a[i]
is a 2D array,
and a[i][j]
is a 1D array. This 3D array can also be accessed
via flat indexing, e.g.,
a[i][j][k] == (&a[0][0][0])[600 * i + 30 * j + k]
Nested views provide a mechanism to access array elements through nested indexing and to extract lower-dimensional “slices” of multidimensional arrays. Nested views are mutable.
Nested views are associated with a dimensionality. For instance,
if v
is a 3D nested view of a 3D array, then v[i]
is a 2D nested view (of a 3D array), v[i][j]
is a 1D nested
view (of a 3D array), and v[i][j][k]
is a scalar array element.
Note that the order of indices is reversed when using nested indexing
compared to multidimensional indexing, e.g.,
v(i, j, k) == v[k][j][i]
.
Whereas operator[]
on an array object accesses an element
through flat indexing, the same array can be accessed through a
nested view to in effect provide nested array indexing:
zfp::array3d a(30, 20, 10, rate); // define 30x20x10 3D array
assert(a[32] == a(2, 1, 0)); // OK: flat and multidimensional indexing
assert(a[32] == a[0][1][2]); // ERROR: a does not support nested indexing
zfp::array3d::nested_view v(&a); // define a nested view of a
assert(a[32] == v[0][1][2]); // OK: v supports nested indexing
zfp::array2d b(v[5]); // define and deep copy 30x20 2D slice of a
assert(a(2, 1, 5) == b(2, 1)); // OK: multidimensional indexing
-
class
array2::
nested_view1
¶ View of a 1D slice of a 2D array.
-
class
array2::
nested_view2
¶ 2D view of a 2D (sub)array.
-
class
array3::
nested_view1
¶ View of a 1D slice of a 3D array.
-
class
array3::
nested_view2
¶ View of a 2D slice of a 3D array.
-
class
array3::
nested_view3
¶ 3D view of a 3D (sub)array.
-
array3::nested_view3::
nested_view3
(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)¶ Whole-array and sub-array nested view constructors. See const_view constructors for details. Lower-dimensional view constructors are not accessible to the user but are invoked when accessing views via nested indexing.
-
array2::nested_view1::
size_x
() const¶
-
array2::nested_view2::
size_x
() const¶
-
array2::nested_view2::
size_y
() const¶
-
array3::nested_view1::
size_x
() const¶
-
array3::nested_view2::
size_x
() const¶
-
array3::nested_view2::
size_y
() const¶
-
array3::nested_view3::
size_x
() const¶
-
array3::nested_view3::
size_y
() const¶
-
array3::nested_view3::
size_z
() const¶ View dimensions.
-
array3::nested_view2
array3::nested_view3::
operator[]
(uint index) const¶ Return view to a 2D slice of a 3D array.
-
array2::nested_view1
array2::nested_view2::
operator[]
(uint index) const¶
-
array3::nested_view1
array3::nested_view2::
operator[]
(uint index) const¶ Return view to a 1D slice of a 2D or 3D array.
-
Scalar
array2::nested_view1::
operator[]
(uint index) const¶
-
Scalar
array3::nested_view1::
operator[]
(uint index) const¶ Return scalar element of a 2D or 3D array.
-
array3::reference
array3::nested_view1::
operator[]
(uint index)¶ Return reference to a scalar element of a 2D or 3D array.
-
Scalar
array2::nested_view1::
operator()
(uint i) const¶
-
Scalar
array2::nested_view2::
operator()
(uint i, uint j) const¶
-
Scalar
array3::nested_view1::
operator()
(uint i) const¶
-
Scalar
array3::nested_view2::
operator()
(uint i, uint j) const¶
-
Scalar
array3::nested_view3::
operator()
(uint i, uint j, uint k) const¶ Return scalar element of a 2D or 3D array.
Slicing¶
Arrays can be constructed as deep copies of slices of higher-dimensional
arrays, as the code example above shows (i.e.,
zfp::array2d b(v[5]);
). Unlike views, which have reference
semantics, such array slicing has value semantics. In this example,
2D array b is initialized as a (deep) copy of a slice of 3D array a
via nested view v. Subsequent modifications of b have no effect on
a.
Slicing is implemented as array constructors templated on views.
Upon initialization, elements are copied one at a time from the view
via multidimensional indexing, e.g., v(i, j, k)
. Note that
view and array dimensionalities must match, but aside from this an
array may be constructed from any view.
Slicing needs not change the dimensionality, but can be used to copy an equidimensional subset of one array to another array, as in this example:
zfp::array3d a(30, 20, 10, rate);
zfp::array3d::const_view v(&a, 1, 2, 3, 4, 5, 6);
zfp::array3d b(v);
assert(b(0, 0, 0) == a(1, 2, 3));
assert(b.size_x() == 4);
assert(b.size_y() == 5);
assert(b.size_z() == 6);
Slicing adds the following templated array constructors.
-
template<class
View
>array3::
array3
(const View &v)¶ Construct array from a view via a deep copy. The view, v, must support multidimensional indexing. The rate for the constructed array is initialized to the rate of the array associated with the view. Note that the actual rate may differ if the constructed array is a lower-dimensional slice of a higher-dimensional array due to lower rate granularity (see FAQ #12). The cache size of the constructed array is set to the default size.
Private immutable view¶
zfp’s compressed arrays are in general not thread-safe. The main reason for this is that each array maintains its own cache of uncompressed blocks. Race conditions on the cache would occur unless it were locked upon each and every array access, which would have a prohibitive performance cost.
To ensure thread-safe access, zfp provides private mutable and
immutable views of arrays that maintain their own private caches.
The private_const_view
immutable view provides read-only
access to the underlying array. It is similar to a
const_view in this sense, but differs in
that it maintains its own private cache rather than sharing the
cache owned by the array. Multiple threads may thus access the
same array in parallel through their own private views.
Note
Private views do not guarantee cache coherence. If, for example, the array is modified, then already cached data in a private view is not automatically updated. It is up to the user to ensure cache coherence by flushing (compressing modified blocks) or clearing (emptying) caches when appropriate.
The cache associated with a private view can be manipulated in the same way an array’s cache can. For instance, the user may set the cache size on a per-view basis.
Unlike with private mutable views, private immutable views may freely access any element in the array visible through the view, i.e., multiple threads may read the same array element simultaneously. For an example of how to use private views for both read and write multithreaded access, see the diffusion code example.
Private views support only multidimensional indexing, i.e., they are neither flat nor nested.
-
class
array1::
private_const_view
¶
-
class
array2::
private_const_view
¶
-
class
array3::
private_const_view
¶ Immutable views of 1D, 2D, and 3D arrays with private caches.
-
array3::private_const_view::
private_const_view
(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)¶ Whole-array and sub-array private immutable view constructors. See const_view constructors for details.
-
array1::private_const_view::
size_x
() const¶
-
array2::private_const_view::
size_x
() const¶
-
array2::private_const_view::
size_y
() const¶
-
array3::private_const_view::
size_x
() const¶
-
array3::private_const_view::
size_y
() const¶
-
array3::private_const_view::
size_z
() const¶ View dimensions.
-
Scalar
array1::private_const_view::
operator()
(uint i) const¶
-
Scalar
array2::private_const_view::
operator()
(uint i, uint j) const¶
-
Scalar
array3::private_const_view::
operator()
(uint i, uint j, uint k) const¶ Return scalar element of a 1D, 2D, or 3D array.
The following functions are common among all dimensionalities:
-
size_t
arrayANY::private_const_view::
cache_size
() const¶
-
void
arrayANY::private_const_view::
set_cache_size
(size_t csize)¶
Private mutable view¶
The mutable private_view
supports both read and write access
and is backed by a private cache. Because block compression, as needed
to support write access, is not an atomic operation, mutable views
and multithreading imply potential race conditions on the compressed
blocks stored by an array. Although locking the array or individual
blocks upon compression would be a potential solution, this would either
serialize compression, thus hurting performance, or add a possibly large
memory overhead by maintaining a lock with each block.
Note
To avoid multiple threads simultaneously compressing the same block, private mutable views of an array must reference disjoint, block-aligned subarrays for thread-safe access. Each block of 4d array elements must be associated with at most one private mutable view, and therefore these views must reference non-overlapping rectangular subsets that are aligned on block boundaries, except possibly for partial blocks on the array boundary. (Expert users may alternatively ensure serialization of block compression calls and cache coherence in other ways, in which case overlapping private views may be permitted.)
Aside from this requirement, the user may partition the array into
disjoint views in whatever manner is suitable for the application.
The private_view
API supplies a very basic partitioner to
facilitate this task, but may not result in optimal partitions or
good load balance.
When multithreaded write access is desired, any direct accesses to the array itself (i.e., not through a view) could invoke compression. Even a read access may trigger compression if a modified block is evicted from the cache. Hence, such direct array accesses should be confined to serial code sections when private views are used.
As with private immutable views, cache coherence is not enforced. Although this is less of an issue for private mutable views due to the requirement that views may not overlap, each private mutable view overlaps an index space with the underlying array whose cache is not automatically synchronized with the view’s private cache. See the diffusion for an example of how to enforce cache coherence with mutable and immutable private views.
The private_view
class inherits all public functions from
private_const_view
.
-
class
array1::
private_view
: public array1::private_const_view¶
-
class
array2::
private_view
: public array2::private_const_view¶
-
class
array3::
private_view
: public array3::private_const_view¶ Mutable views of 1D, 2D, and 3D arrays with private caches.
-
class
array1::private_view::
view_reference
¶
-
class
array2::private_view::
view_reference
¶
-
class
array3::private_view::
view_reference
¶ Proxy references to array elements specialized for mutable private views.
-
array3::private_view::
private_view
(array3 *array, uint x, uint y, uint z, uint nx, uint ny, uint nz)¶ Whole-array and sub-array private mutable view constructors. See const_view constructors for details.
-
array1::private_view::view_reference
array1::private_view::
operator()
(uint i) const¶
-
array2::private_view::view_reference
array2::private_view::
operator()
(uint i, uint j) const¶
-
array3::private_view::view_reference
array3::private_view::
operator()
(uint i, uint j, uint k) const¶ Return reference to a scalar element of a 1D, 2D, or 3D array.
The following functions are common among all dimensionalities:
-
void
arrayANY::private_view::
partition
(uint index, uint count)¶ Partition the current view into count roughly equal-size pieces along the view’s longest dimension and modify the view’s extents to match the piece indexed by index, with 0 ≤ index < count. These functions may be called multiple times, e.g., to recursively partition along different dimensions. The partitioner does not generate new views; it merely modifies the current values of the view’s offsets and dimensions. Note that this may result in empty views whose dimensions are zero, e.g., if there are more pieces than blocks along a dimension.
-
void
arrayANY::private_view::
flush_cache
() const¶ Flush cache by compressing any modified blocks and emptying the cache.
C bindings¶
zfp 0.5.4 adds cfp: C language bindings for compressed arrays via wrappers around the C++ classes.
The C API has been designed to facilitate working with compressed arrays
without the benefits of C++ operator overloading and self-aware objects,
which greatly simplify the syntax. Whereas one possible design considered
is to map each C++ method to a C function with a prefix, such as
zfp_array3d_get(a, i, j, k)
in place of a(i, j, k)
for
accessing an element of a 3D array of doubles, such code would quickly
become unwieldy when part of longer expressions.
Instead, cfp uses the notion of nested C namespaces that are structs
of function pointers, such as cfp.array3d
. Although this may
seem no more concise than a design based on prefixes, the user may alias
these namespaces (somewhat similar to C++ using namespace
declarations) using far shorter names via C macros or local variables.
For instance:
const cfp_array3d_api _ = cfp.array3d; // _ is a namespace alias
cfp_array3d* a = _.ctor(nx, ny, nz, rate, 0, 0);
double value = _.get(a, i, j, k);
_.set(a, i, j, k, value + 1);
which is a substitute for the C++ code
zfp::array3d a(nx, ny, nz, rate, 0, 0);
double value = a(i, j, k);
a(i, j, k) = value + 1;
Because the underlying C++ objects have no corresponding C representation,
and because C objects are not self aware (they have no implicit this
pointer), the C interface interacts with compressed arrays through
pointers to opaque types that cfp converts to pointers to
the corresponding C++ objects. As a consequence, cfp compressed arrays
must be allocated on the heap and explicitly freed via designated
destructor functions to avoid memory leaks. The C++ constructors
are mapped to C by allocating objects via C++ new
. Moreover, the
C API requires passing an array self pointer in order to manipulate the
array.
As with the C++ classes, array elements can be
accessed via multidimensional array indexing, e.g., get(array, i, j)
,
and via flat, linear indexing, e.g., get_flat(array, i + nx * j)
.
-
cfp_array1f
¶
-
cfp_array2f
¶
-
cfp_array3f
¶
-
cfp_array1d
¶
-
cfp_array2d
¶
-
cfp_array3d
¶ Opaque types for 1D, 2D, and 3D compressed arrays of floats and doubles.
-
cfp_array1f*
cfp_array1f.ctor
(uint nx, double rate, const float* p, size_t csize)¶
-
cfp_array2d*
cfp_array1d.ctor
(uint nx, double rate, const double* p, size_t csize)¶
-
cfp_array2f*
cfp_array2f.ctor
(uint nx, uint ny, double rate, const float* p, size_t csize)¶
-
cfp_array2d*
cfp_array2d.ctor
(uint nx, uint ny, double rate, const double* p, size_t csize)¶
-
cfp_array3f*
cfp_array3f.ctor
(uint nx, uint ny, uint nz, double rate, const float* p, size_t csize)¶
-
cfp_array3d*
cfp_array3d.ctor
(uint nx, uint ny, uint nz, double rate, const double* p, size_t csize)¶ Array constructors. If p is not
NULL
, then the array is initialized from uncompressed storage; otherwise the array is zero initialized. csize is the minimum size cache (in bytes) to use. If csize is zero, a default size is chosen.
-
float
cfp_array1f.get
(const cfp_array1f* a, uint i)¶
-
float
cfp_array2f.get
(const cfp_array2f* a, uint i, uint j)¶
-
float
cfp_array3f.get
(const cfp_array3f* a, uint i, uint j, uint k)¶
-
double
cfp_array1d.get
(const cfp_array1d* a, uint i)¶
-
double
cfp_array2d.get
(const cfp_array2d* a, uint i, uint j)¶
-
double
cfp_array3d.get
(const cfp_array3d* a, uint i, uint j, uint k)¶ Array accessors via multidimensional indexing.
-
void
cfp_array1f.set
(const cfp_array1f* a, uint i, float val)¶
-
void
cfp_array2f.set
(const cfp_array2f* a, uint i, uint j, float val)¶
-
void
cfp_array3f.set
(const cfp_array3f* a, uint i, uint j, uint k, float val)¶
-
void
cfp_array1d.set
(const cfp_array1d* a, uint i, double val)¶
-
void
cfp_array2d.set
(const cfp_array2d* a, uint i, uint j, double val)¶
-
void
cfp_array3d.set
(const cfp_array3d* a, uint i, uint j, uint k, double val)¶ Array mutators for assigning values to array elements via multidimensional indexing.
-
void
cfp_array1f.get_array
(const cfp_array1f* self, float* p)¶
-
void
cfp_array1d.get_array
(const cfp_array1d* self, double* p)¶
-
void
cfp_array2f.get_array
(const cfp_array2f* self, float* p)¶
-
void
cfp_array2d.get_array
(const cfp_array2d* self, double* p)¶
-
void
cfp_array3f.get_array
(const cfp_array3f* self, float* p)¶
-
void
cfp_array3d.get_array
(const cfp_array3d* self, double* p)¶ Decompress entire array; see
array::get()
.
-
void
cfp_array1f.set_array
(cfp_array1f* self, const float* p)¶
-
void
cfp_array1d.set_array
(cfp_array1d* self, const double* p)¶
-
void
cfp_array2f.set_array
(cfp_array2f* self, const float* p)¶
-
void
cfp_array2d.set_array
(cfp_array2d* self, const double* p)¶
-
void
cfp_array3f.set_array
(cfp_array3f* self, const float* p)¶
-
void
cfp_array3d.set_array
(cfp_array3d* self, const double* p)¶ Initialize entire array; see
array::set()
.
-
uint
cfp_array2f.size_x
(const cfp_array2f* self)¶
-
uint
cfp_array2f.size_y
(const cfp_array2f* self)¶
-
uint
cfp_array3f.size_x
(const cfp_array3f* self)¶
-
uint
cfp_array3f.size_y
(const cfp_array3f* self)¶
-
uint
cfp_array3f.size_z
(const cfp_array3f* self)¶
-
uint
cfp_array2d.size_x
(const cfp_array2d* self)¶
-
uint
cfp_array2d.size_y
(const cfp_array2d* self)¶
-
uint
cfp_array3d.size_x
(const cfp_array3d* self)¶
-
uint
cfp_array3d.size_y
(const cfp_array3d* self)¶
-
uint
cfp_array3d.size_z
(const cfp_array3d* self)¶
-
void
cfp_array1f.resize
(cfp_array1f* self, uint n, int clear)¶
-
void
cfp_array2f.resize
(cfp_array2f* self, uint nx, uint ny, int clear)¶
-
void
cfp_array3f.resize
(cfp_array3f* self, uint nx, uint ny, uint nz, int clear)¶
-
void
cfp_array1d.resize
(cfp_array1d* self, uint n, int clear)¶
-
void
cfp_array2d.resize
(cfp_array2d* self, uint nx, uint ny, int clear)¶
-
void
cfp_array3d.resize
(cfp_array3d* self, uint nx, uint ny, uint nz, int clear)¶
The six array types share many functions that have the same signature.
Below, each instance of cfp_array
generically refers to one of
those six types.
-
cfp_array*
cfp_array.ctor_default
()¶ Default constructor. Allocate an empty array that later can be resized and whose rate and cache size can be set by
cfp_array.set_rate()
andcfp_array.set_cache_size()
. Return a pointer to the constructed array.
-
cfp_array*
cfp_array.ctor_copy
(const cfp_array* src)¶ Copy constructor. Return a pointer to the constructed array.
-
void
cfp_array.dtor
(cfp_array* self)¶ Destructor. The destructor not only deallocates any compressed data owned by the array, but also frees memory for itself, invalidating the self pointer upon return. Note that the user must explicitly call the destructor to avoid memory leaks.
-
void
cfp_array.deep_copy
(cfp_array* self, const cfp_array* src)¶ Perform a deep copy of src analogous to the C++ assignment operator.
-
double
cfp_array.rate
(const cfp_array* self)¶ See
array::rate()
.
-
double
cfp_array.set_rate
(cfp_array* self, double rate)¶ See
array::set_rate()
.
-
size_t
cfp_array.cache_size
(const cfp_array* self)¶ See
array::cache_size()
.
-
void
cfp_array.set_cache_size
(cfp_array* self, size_t csize)¶
-
void
cfp_array.clear_cache
(const cfp_array* self)¶ See
array::clear_cache()
.
-
void
cfp_array.flush_cache
(const cfp_array* self)¶ See
array::flush_cache()
.
-
size_t
cfp_array.compressed_size
(const cfp_array* self)¶
-
uchar*
cfp_array.compressed_data
(const cfp_array* self)¶
-
size_t
cfp_array.size
(const cfp_array* self)¶ See
array::size()
.
-
float
cfp_array.get_flat
(const cfp_array* a, uint index)¶
-
double
cfp_array.get_flat
(const cfp_array* a, uint index) Flat index array accessors; see
array::operator[]()
.
-
void
cfp_array.set_flat
(const cfp_array* a, uint index, float val)¶
-
void
cfp_array.set_flat
(const cfp_array* a, uint index, double val) Flat index array mutators; set array element with flat index to val.