Controlling Array properties with Descriptors

The ILNumerics Array Visualizer is a powerful live data plotting and visualization tool for Visual Studio debug sessions and available for many popular languages. It allows to visualize arrays of arbitrary dimensionality and element type. Supported array types range from common 1D double[] System.Arrays in .NET, arrays of derived types in FORTRAN to void* pointers in C. General documentation of the Array Visualizer is found here. This section deals with advanced options to control the way your memory data are interpreted via array descriptors.

Read on this page: 

• Two modes for the ILNumerics Array Visualizer
• When to use array descriptors? 
• Array descriptor format specification
• Array descriptor usage examples
  • Fun visualizing complex[] arrays (C#) 
  • Debugging structs & recordsets (Visual Basic)
  • Reinterpreting the Element Type (Visual Basic, cont.)
  • Visualizing multiple Variables (Visual Basic, cont.)
  • Plotting Pointers and Addresses (C/C++)
  • Plotting Image Data (C/C++)
• Sources for Array Property Meta Data 
  • Locking Array Descriptor Components
  • Unlocking Array Descriptor Components

Two Modes for the Array Visualizer

In order to extract and visualize array data from memory the Array Visualizer requires the following info:

• The type of the elements,
• Number and lengths of the array dimensions,
• The starting address in memory and
• The storage format (row- vers. column major order / element strides for each dimension).

The Array Visualizer will collect all available information automatically. If it is not possible to acquire a needed piece of information the user is asked for more details. This is where array descriptors come into play. Array descriptors provide a way to let the developer specify certain array properties by use of a simple, yet flexible descriptor language.

The expressions are entered in one of the two available modes:

1. Array descriptor disabled: No array descriptors are displayed (default).
2. Array descriptor enabled: The array expression is prefixed by an array descriptor pattern.

The mode is toogled via "Show Array Descriptor" from the top right menu of the Array Visualizer Tool Window:

In order to access and control the properties of an array, the "Show Array Descriptor" mode must be

enabled. Note, that the tool window activates this mode automatically if the array lacks meta information.

When to use Array Descriptors?

Next to common array objects the Array Visualizer is used to describe, reinterpret and reshape arbitrary memory segments and to plot the values in a graphical way. The array descriptor supports the user by defining extended properties of the array. Here, the term 'array' may or may not correspond to a concrete array instance. The descriptor provides the information on how to interprete your debug memory in a way which makes it useful and suitable for visualization. It allows to explicitly define any array property.

Array descriptors come in handy in the following situations:

• Visualizing from plain memory addresses.
• Plotting arrays of structs / recordsets. E.g.: plotting the real parts of complex[], see below.
• Reshaping N-D arrays from memory, including slicing and subarray plotting.
• Handling array data which were created in foreign storage schemes (row major / column major). 
• Plotting data referenced via pointer variables.

Array Descriptor Format

                   {Type[Dimensions]StridesOffset}

Where

• Type [required] must be any valid predefined numeric element type in the current language. For C#, the list includes double, int, short, long,... For Visual Basic Single, Double, and Integer are common examples. In C/C++ unsigned long, and long long would be some valid TYPE patterns. Consult the language specific documentation for a more comprehensive list of supported element types.
• Dimensions [required] specifies the list of dimension lengths. Positive integer(s) or 0 (zero). If the array assembles more than one dimension Dimensions is a comma separated list of integers.
• Strides [optional] specifies the storage format, i.e.: the spacing in memory between individual elements within each dimension. The strides specifier can be one of:
  • The single character 'R' for row-major storage. This is the default for all .NET System.Arrays and C arrays.
  • The single character 'C' for column-major storage. This is the default for all FORTRAN arrays.
  • A comma separated list of unsigned integer values. The number of integers provided must match the number of dimensions specified by Dimensions. The list must be enclosed in '[ ]' brackets. Ex.: [1,5] without any white space characters. Integers can define arbitrary spacings between elements for each dimension. The unit for each dimension is elements (just the integer) or bytes (integer, prefixed by; 'b'). Ex.: [8b,4,702b] defines 8 bytes spacing along the 1st dimension, 4 elements spacing along the second dimension and 702 bytes spacing along the 3rd dimension. 
• Offset [optional] is a single integer number, optionally signed. It shifts the starting address of the array by the number of elements of Type. Offset may end with the character 'b'. In this case the unit of the offset is bytes instead of elements. Negative offsets are allowed. The default offset (0) is omitted.

Examples of valid array descriptor expressions:

• {int[3]R} – 1D int32 array, 3 elements, row major storage layout.
• {double[1000,20,30]} – 3D double array, 1000 x 20 x 30. language dependant storage layout.
• {Integer[100,200,30]C} – 3D Int32 array in Visual Basic, 100 x 200 x 30. Providing C as storage layout overrides the language default value and considers this array as stored in column major order.
• {short[10,20][2,20b]40b} – 2D short array in C#, 10 x 20 elements, elements start is offset by 40 bytes. Elements in dimension 1 are spaced by 2 short elements (4 bytes) apart. In the second dimension element spacing is 20 bytes.
• {Single[10,20]R-40b} – 2D float array in Visual Basic, 10 x 20 elements, elements start is offsetted by -40 bytes.
• {short[10000]C100} – 1D C# array with 10.000 elements, starting with the 100th element.
• {char[100,100][4,100]} – visualizing interleaved elements from a 400x100 bytes memory block in C/C++.

Array Descriptor Usage

The array descriptor gives complete control over how your data is read from memory. It allows to reshape / re-orient arrays, to slice parts from arrays, transpose data, reinterpret the element type – all what is required to bring the data into a shape, suitable for plotting. In this section we give a general (mixed-language) overview of handling array descriptors. Consult the language specific documentation for more details: C#, F#, Visual Basic, C/C++, FORTRAN.

C# Example: Array descriptors for Full DEBUG DATA control

Consider the following C# array definition. It creates an array of 20 complex values as a row vector.

In the following we are going to use this System.Array to learn some useful visualizations. By default, no array descriptor is used when we enter the name of the variable into the Array Visualizer expression text box:

The text output gives the best details here. While other options (bar plots, scatter plots) are available, they are not able to handle complex numbers and would utilize the real parts only. At first, let's reshape the extracted array to form a column vector instead of a row vector. We enable the array descriptor mode from the menu:

The descriptor shows all details which are known for the array: the element type, the number and length of its dimensions and the storage layout – .NET arrays are stored in row major order as is indicated by "R".

We can now go ahead and change the array properties arbitrarily. In order to create a column vector we just have to flip the dimensions in the array descriptor. From [1,20] to [20,1]:

Now the array is fully visible without scrolling in the Array Visualizer tool window.

Next we want to extract real and imaginary data as individual elements. Recall the memory layout of an array of complex values. Real and imaginary parts are stored interleaved in memory, starting with the real part of the first complex number:

In order to plot the real part of the array we need to modify the array descriptor to form a vector of double elements of length 20. The spacing between individual elements (strides) is 2 double elements. So we change the array descriptor from:

to:

.

Here, the "R" (row major storage layout, as read from actual array meta data) was replaced by a more detailed strides expression. "[2,1]" means: elements in the first dimension are separated by a memory offset corresponding to the size of 2 elements. In the second dimension adjacent elements are found by computing the offset based on a single element ("1"). The resulting double[] array is suitable for plotting the real part:

What if we want the imaginary part instead? This can be achieved by utilizing of the 'OFFSET' component of the array descriptor. The required vector of double[20] must start 8 bytes later, which corresponds to the length of 1 element:

Note, that more fine grained control over the offset value is possible by defining the offset unit as 'bytes': 

This extracts a plot of the imaginary part: 

Finally, array descriptors can be used to combine both: real and imaginary parts together to form an array which is directly suitable for plotting XY line plots, surface plots and combined bar plots in higher numbers of dimensions. Next we are going to transform the interleaved array of stucts into a double matrix of real values in the first row and imaginary values as the second row. 

Steps needed:

1. Select the rank, dimension lengths and element type: the target array ought to be a 2 x 20 matrix of Double elements.
2. Choose the spacing between individual elements in each dimension of the matrix. The 1st dimension stores the real and imaginary values; real values come first, imaginary values next. Both are stored subsequentially in memory, hence have a spacing of 1 element. The 2nd dimension corresponds to each complex number instance / elements of complex_sys array. Individual elements of the complex_sys array are stored with a spacing of 16 bytes, or 2 double elements apart from each other. The needed strides are: [1,2]

The full array descriptor + array expression therefore becomes:

Note how the matrix form of the complex array immediately enables the full set of plotting options. Next to regular text output the 2D matrix of complex data can be visualized by the following chart types:

XY-Line Plot	Scatter Plot	Rows Plot
Surface Plot	ImageSC Plot	Bar Plot