ILNumerics - Technical Computing

Modern High Performance Tools for Technical

Computing and Visualization in Industry and Science

 

Refining Array properties with Descriptors

The ILNumerics Array Visualizer is a flexible live data plotting and visualization tool for Visual Studio debug sessions and available for many popular languages. It allows to visualize arrays of arbitrary dimensionality and element type. Supported array types range from common 1D double[] System.Arrays in .NET, arrays of derived types in FORTRAN to void* pointers in C. General documentation of the Array Visualizer is found here. This section deals with advanced options to control the way your memory data are interpreted via array descriptors.

Two Modes for the Array Visualizer

In order to extract and visualize array data from memory the Array Visualizer requires the following info:

  • The type of the elements,
  • Number and lengths of the array dimensions,
  • The starting address in memory and
  • The storage format (row- vers. column major order / element strides for each dimension).

The Array Visualizer will collect all available information automatically. If it is not possible to acquire a needed piece of information the user is asked for more details. This is where array descriptors come into play. Array descriptors provide a way to let the developer specify certain array properties by use of a simple, yet flexible descriptor language.

The expressions are entered in one of the two available modes:

  1. Array descriptor disabled: No array descriptors are displayed (default).
  2. Array descriptor enabled: The array expression must be prepended by an array descriptor pattern.

The mode is toogled via "Use Array Descriptors" from the top right menu of the Array Visualizer Tool Window. In order to access and control the properties of an array, the "Use Array Descriptors" mode must be enabled. The tool window activates this mode automatically if it needs more details on an array.

When to use Array Descriptors?

Next to common array objects the Array Visualizer is used to describe, reinterpret and reshape arbitrary memory segments and to plot the values in a graphical way. The array descriptor supports the user by defining extended properties of the array. Here, the term 'array' may or may not correspond to a concrete array instance. The descriptor provides the information on how to interprete your debug memory in a way which makes it useful and suitable for visualization. It allows to explicitly define any array property.

Array descriptors come in handy in the following situations:

  • Visualizing from plain memory addresses.
  • Plotting arrays of structs / recordsets. E.g.: plotting the real parts of complex[], see below.
  • Reshaping N-D arrays from memory, including slicing and subarray plotting.
  • Handling array data which were created in foreign storage schemes (row major / column major). 
  • Plotting data referenced via pointer variables.

Array Descriptor Format

                   {TYPE[DIMS]STRIDES+OFFSET}

Where

  • TYPE [required] must be any valid predefined numeric element type in the current language. For C#, the list includes doubleint, short, long,... For Visual Basic Single, Double, and Integer are common examples. In C/C++ unsigned long, and long long would be some valid TYPE patterns. Consult the language specific documentation for a more comprehensive list of supported element types.
  • DIMS [required] specify the list of dimension lengths. If the array assembles more than one dimension DIMS is a comma separated list of integers.
  • STRIDES [optional] specifies the storage format, i.e.: the spacing in memory between individual elements within each dimension. The strides specifier can be any one of:
    • The single character 'R' for row-major storage. This is the default for all .NET System.Arrays and C arrays.
    • The single character 'C' for column-major storage. This is the default for all ILNumerics and FORTRAN arrays.
    • A comma separated list of unsigned integer values. The number of integers provided must match the number of dimensions specified by DIMS. The list must be enclosed in '[ ]' brackets. Ex.: [1,5]
  • OFFSET [optional] is an integer number, optionally signed. It shifts the starting address of the array by the number of elements of TYPE. If OFFSET ends with 'b' then the number specifies the number of bytes to shift the starting address instead. Negative offsets are allowed. Default offset is 0.

Examples of valid array descriptor expressions:

  • {int[3]R} – 1D int32 array, 3 elements, row major storage layout.
  • {double[1000,20,30]} – 3D double array, 1000 x 20 x 30. language dependant storage layout.
  • {Integer[100,200,30]C} – 3D Int32 array in Visual Basic, 100 x 200 x 30. Providing C as storage layout overrides the language default value and considers this array as stored in column major order.
  • {short[10,20][2,20b]40b} – 2D short array in C#, 10 x 20 elements, elements start is offset by 40 bytes. Elements in dimension 1 are spaced by 2 short elements (4 bytes) apart. In the second dimension element spacing is 20 bytes.
  • {Single[10,20]R-40b} – 2D float array in Visual Basic, 10 x 20 elements, elements start is offsetted by -40 bytes.
  • {short[10000]C100} – 1D C# array with 10.000 elements, starting with the 100th element.
  • {char[100,100][4,100]} – visualizing interleaved elements from a 400x100 bytes memory block in C/C++.

Examples - Array Descriptor Usage

The array descriptor gives great control over how your data is read from memory. It allows to reshape / re-orient arrays for visualization, allow to slice parts from arrays, transpose data, reinterpret the array element type and much more. In this section we give a general (mixed-language) overview of handling array descriptors. Consult the language specific documentation for more details: C#, F#, Visual Basic, C/C++, FORTRAN.

C# Example: Array descriptors for DEBUG DATA type & shape control

Consider the following C# array definition. It creates an array of 20 complex values as a row vector.

We are going to use this array in the following to work out some useful visualizations. By default, no array descriptor is used when we enter the name of the variable into the Array Visualizer expression text box:

The text output gives the best details here. While other options (bar plots, scatter plots) are available, they are not able to handle complex numbers and would utilize the real parts only. At first, let's reshape the extracted array to form a column vector instead of a row vector. We enable the array descriptor mode from the menu:

The descriptor shows all details which are known of the array: the element type, the number and length of its dimensions and the storage layout – .NET arrays are stored in row major order as indicated by "R".

We can now go ahead and change the array properties arbitrarily. In order to create a column vector we just have to flip the dimensions in the array descriptor. From [1,20] to [20,1]:

Now the array is fully visible without scrolling in the Array Visualizer tool window.

Next we want to extract real and imaginary data as individual elements. Recall the memory layout of an array of complex values. Real and imaginary parts are stored interleaved in memory, starting with the real part of the first complex number:

In order to plot the real part of the array we need to modify the array descriptor to form a vector of double elements of length 20. The spacing between individual elements (strides) is 2 double elements. So we change the array descriptor from

,

to

.

The double[] array we get is suitable for plotting the real part:

What if we would need the imaginary part instead? This can be achieved by utilization of the 'OFFSET' component of the array descriptor. The required vector of double[20] would simply have to start 8 bytes later, which corresponds to the length of 1 element:

Learn how to achieve the above more easily

Plotting the real and imaginary parts by handling strides and offsets is only one option for plotting complex data as in this example. The same could be achieved without using array descriptors at all! Learn how in the language specific tutorials for C# and Visual Basic. The purpose of this manual is to show the options of the array descriptor. Moreover, while easier ways exist, it is highly recommended to become familiar with the array descriptor since this brings the most flexibility in advanced visualization scenarios.

 

 

Finally, array descriptors can be used to combine both: real and imaginary parts together to form an array which is directly suitable for plotting XY line plots, surface plots and combined bar plots in higher numbers of dimensions. Next we are going to transform the interleaved array of stucts into a double matrix of real values in the first row and imaginary values as the second row. 

Steps needed:

  1. Select the rank, dimension lengths and element type: the target array ought to be a 2 x 20 matrix of Double elements.
  2. Choose the spacing between individual elements in each dimension of the matrix. The 1st dimension stores the real and imaginary values; real values come first, imaginary values next. Both are stored subsequentially in memory, hence have a spacing of 1 element. The 2nd dimension corresponds to each complex number instance / elements of complex_sys array. Individual elements of the complex_sys array are stored with a spacing of 16 bytes, or 2 double elements apart from each other. So the needed strides are: [1,2]

The full array descriptor + array expression therefore becomes:

Note how the matrix form of the complex array immediately enables the full set of plotting options. Next to text output the 2D matrix of complex data can be visualized by the following chart types:

XY-Line Plot Scatter Plot Rows Plot
Surface Plot ImageSC Plot Bar Plot

Read more about: 

Visual Basic Example: Handling arrays of user defined data types / structs / recordsets.

Array of custom structs / recordsets can easily and flexible be plotted by picking individual elements / fields from the structs and having the Array Visualizer extract the corresponding values from all struct instances within the array. The next example will make this clear.

At first a custom structure Person is created. It holds a string variable and 3 Integer variables each. The storage layout was explicitly defined, which simplifies things but is not obligatory in general. Next an array of Persons is created which forms the presidents of the United States, ordered by their heights. These heights can easily be plotted without stressing the Array Visualizer. Just stop at some debug position and enter 'presidents(0).Height' into the Array Visualizer:

We could use the same concept to plot any data member of the struct by exchanging its name for the 'Height' term in the current expression. However, here we want to play with advanced array descriptor options. Let's change the expression to read 'presidents(3)' and press ENTER.

It is worth to note 3 things here:

  1. The array descriptor was enabled automatically, since we are required to explicitly enter the element type we want to plot. Expect this to happen whenever array property data for plotting are missing. You will find the missing component selected in the array descriptor expression, ready for you to enter the missing detail.
  2. The new array descriptor exposes some predefined data: the rank of the array (1), the length of the only dimension [41] and the spacing / stride for elements in this dimension [16 bytes]. All those data have been extracted automatically from the entered expression and by help of the debug engine. 
  3. The third noteworthy info relates to the number of presidents existing. By time of writing, Barack Obama is the 44th US president. 44 presidents have been defined in the Presidents array. However, the length of the only dimension as predefined by the array descriptor is 41!? Where are the missing elements? The answer: they are to be excluded from the extracted array. The reason is that we had entered an expression pointing to the 4th element (index 3). The array descriptor recognizes that there are 44 struct elements in the array but subtracts 3 from it to start with the 4th element without reaching over the end of the array.

Redefining Element Types

Continuing the example we enter UInteger as element type. We know that all numbers we have stored are positive. So we can be sure that unsigned integer is a matching type. The Array Visualizer will not check for it! It will simply believe in you that you know what you are doing. Failing to do so will cause useless values at best. Here, we had defined the integer fields of the struct as signed integers. However, they indeed are compatible and we will get the same values as if we had choosen Integer instead. 

Just like for complex numbers it is possible to combine multiple data from the struct into the same extraction array for plotting. We could plot the Year element against the Height values. Therefore, we combine the corresponding fields from the structs into a matrix of size 2 x 44:

Steps needed:

  • Start as if plotting the Number fields. Enter 'presidents(0).Year' into the textbox. We choose 'Year' because this field comes first of the Height field.
  • If you have modified the existing expression without clearing the whole text box first make sure to hold down the CTRL key when pressing ENTER. This will reset the array descriptor and re-evaluate the properties of the new array. Otherwise an existing array descriptor takes precedence over the array expression.
  • We receive a column vector of all Year fields. The array descriptor reads: {Integer[44][16b]4b}presidents(0).Number
  • Combining data mostly means: adding a new dimension to the extracted array and to specify the strides for the new data. This is done by changing the array descriptor: We have added a new dimension of length 2 (Year + Height = 2 Integers) and defined the strides for them. The Height field comes directly after the Year field. Therefore the stride is 1. With the newly extracted matrix we can choose from the full spectra of plotting types. Let's plot the heights data against the Years by selecting the scattered data plot: