Importing & Exporting Array Data
This section deals with functions for importing data into and exporting data from ILNumerics arrays. Import and export of array elements is important for convenient creation of arrays and for interacting with external, managed and unmanged APIs.
This information underwent significant changes in version 5.
In order to import data from system arrays
Note that ILNumerics will make a copy of the values from the System.Array provided and will not reference the original elements afterwards. (This is a difference to earlier versions of ILNumerics!)
The array member function
The ExportValues(ref T, StorageOrders order) overload allows to control the order of the elements for export.
The use of the functionality described in this section requires some expert knowledge of the internal workings of ILNumerics arrays and should be used with care! In order to alter elements of ILNumerics arrays safely, use the common subarray functionality.
ILNumerics arrays store numeric data on unmanaged heaps. In the most common situation, arrays use memory allocated from the process' virtual memory manager and reuse them efficiently for subsequent array creations. With version 5 things became even more interesting: there is no guarantee about the memory location anymore! It might as well be the case, that elements are (exclusively) stored on some accelerating device or that elements may not even exist as concrete instances. Even though ILNumerics will warn you before accessing any memory in a wrong way, this is one aspect users have to consider when attempting to access internal data on a memory / pointer level, nevertheless.
On the other hand, going down to low level pointer access can be a useful performance gain and became much more feasible with version 5. While in version 4 ILNumerics arrays used managed 1D arrays which had to be pinned in order to work with pointers, there is no need for pinning in version 5 anymore. The memory locations where array elements are stored are no longer subject of garbage collection. Also, invoking native libraries is now even more efficient.
For completeness and to add some complexity, it must be noted that reference type elements, as for Array<string> and Cell, are still stored on managed heaps in version 5. Their elements can be accessed on a low level, based on 1D system arrays, too. In this case, all common precautions (GC) must be considered.
Experts use the pointer provided by A.GetHostPointerForRead() on an ILNumerics array A in order to access the internal storage for numeric ILNumerics array directly. A.GetHostPointerForRead() gives the address of the first element of A. Keep the following considerations in mind:
- Make sure the pointer is valid. Null pointers indicate storages existing on other devices than the host device. Also, no reference type elements can be accessed this way - they remain on the managed heap. Be prepared to see an InvalidOperationException in this case.
- The pointer can be used for read access only! Reason: the storage may be shared by multiple ILNumerics arrays. Altering elements may lead to unexpected side effects.
- ILNumerics remains responsible for the internal storage. The user must ensure, not to reference the storage after the corresponding array ran out of scope, was changed in any way, reassigned or disposed.
- Be aware, that ILNumerics stores elements in arbitrary order. Use the Size object returned from A.S in order to figure out the strides for each dimension (A.S.GetStride(int)). Or use iterators to translate pointers to element addresses:
A reference for the internal storage array can be acquired for writing purpose also:
Note, that since the pointer returned from GetHostPointerForWrite() can be used to alter the array the function is only available on mutable arrays, namely: Array<T>, OutArray<T>, Logical, OutLogical, Cell, OutCell.
A number of helper functions is available for converting various formats of indices into multidimensional arrays. The following snippet gives an example:
Data Exchange File Formats
ILNumerics offers many ways to exchange data with other applications and APIs. Next to the common options provided by the .NET framework (text / csv, XML, binary streams, etc.) ILNumerics provides the following options to read/write numeric data:
- ILNumerics.MatFile class for handling Matlab *.mat files
- HDF5 as general data exchange format
The Data I/O section details these options.