ILNumerics - Technical Computing Tools

High Performance Framework for Visualization

and Computing in Industry and Science

 

tgt

Importing & Exporting Array Data

This section deals with functions for importing data into and exporting data from ILNumerics arrays. Import and export of array elements is important for convenient creation of arrays and for interacting with external, managed and unmanged APIs.

This information underwent significant changes in version 5. 

Importing System.Array, T[]

In order to import data from system arrays T[] the creation function Math.vector(T[]) is used. Alternatively, the T[] array can simply be assigned to ILNumerics array variables and used as parameters in function expecting ILNumerics arrays.

Note that ILNumerics will make a copy of the values from the System.Array provided and will not reference the original elements afterwards. (This is a difference to earlier versions of ILNumerics!)

Exporting System.Array, T[]

The array member function ExportValues(ref T[]) is used to copy all element values from the ILNumerics array into the system array. If the provided system array is of sufficient length it is filled with a shallow copy of the elements. Otherwise, a new system array is allocated from the managed heap and filled with shallow copies of the elements:

The ExportValues(ref T[], StorageOrders order) overload allows to control the order of the elements for export. 

internal data storage

The use of the functionality described in this section requires some expert knowledge of the internal workings of ILNumerics arrays and should be used with care! In order to alter elements of ILNumerics arrays safely, use the common subarray functionality. 

 

ILNumerics arrays store numeric data on unmanaged heaps. In the most common situation, arrays use memory allocated from the process' virtual memory manager and reuse them efficiently for subsequent array creations. With version 5 things became even more interesting: there is no guarantee about the memory location anymore! It might as well be the case, that elements are (exclusively) stored on some accelerating device or that elements may not even exist as concrete instances. Even though ILNumerics will warn you before accessing any memory in a wrong way, this is one aspect users have to consider when attempting to access internal data on a memory / pointer level, nevertheless.

On the other hand, going down to low level pointer access can be a useful performance gain and became much more feasible with version 5. While in version 4 ILNumerics arrays used managed 1D arrays which had to be pinned in order to work with pointers, there is no need for pinning in version 5 anymore. The memory locations where array elements are stored are no longer subject of garbage collection. Also, invoking native libraries is now even more efficient. 

For completeness and to add some complexity, it must be noted that reference type elements, as for Array<string> and Cell, are still stored on managed heaps in version 5. Their elements can be accessed on a low level, based on 1D system arrays, too. In this case, all common precautions (GC) must be considered.

Read access

Experts use the pointer provided by A.GetHostPointerForRead() on an ILNumerics array A in order to access the internal storage for numeric ILNumerics array directly. A.GetHostPointerForRead() gives the address of the first element of A. Keep the following considerations in mind: 

  1. Make sure the pointer is valid. Null pointers indicate storages existing on other devices than the host device. Also, no reference type elements can be accessed this way - they remain on the managed heap. Be prepared to see an InvalidOperationException in this case.
  2. The pointer can be used for read access only! Reason: the storage may be shared by multiple ILNumerics arrays. Altering elements may lead to unexpected side effects.
  3. ILNumerics remains responsible for the internal storage. The user must ensure, not to reference the storage after the corresponding array ran out of scope, was changed in any way, reassigned or disposed. 
  4. Be aware, that ILNumerics stores elements in arbitrary order. Use the Size object returned from A.S in order to figure out the strides for each dimension (A.S.GetStride(int)). Or use iterators to translate pointers to element addresses: 
Note: overloads exist for GetHostPointerForRead/Write(order), allowing one to reorder the elements into a given order before returning back the pointer to the first element. This reordering is performed only when really necessary. Nevertheless, if performed, it can degrade performance. Consider preventing from such reordering in high performance scenarios.
Note: the GetArrayForRead() function from version 4 now returns a copy of the internally stored elements for numeric arrays on the managed heap and is left for compatibility with older versions and for low-level access to the elements of reference type arrays (Array<string>, Cell).

Write access

A reference for the internal storage array can be acquired for writing purpose also: A.GetHostPointerForWrite() returns the address of the first element for mutable A and ensures that no side effects exist when altering elements. The precautions 1, 3 and 4 from the paragraph Read Access above apply the same.

Note, that since the pointer returned from GetHostPointerForWrite() can be used to alter the array the function is only available on mutable arrays, namely: Array<T>, OutArray<T>, Logical, OutLogical, Cell, OutCell.  

Index handling

A number of helper functions is available for converting various formats of indices into multidimensional arrays.  The following snippet gives an example:

 

Data Exchange File Formats

ILNumerics offers many ways to exchange data with other applications and APIs. Next to the common options provided by the .NET framework (text / csv, XML, binary streams, etc.) ILNumerics provides the following options to read/write numeric data:

The Data I/O section details these options.