Visualization Engine Improvements

The main goal of the GDI renderer is still to provide a fully compatible alternative to the OpenGL renderer. It automatically replaces the OpenGL default renderer if a problem / incompatible hardware etc. was detected at runtime. The focus lays on feature completeness and precision of the rendering result. The quality of the GDI renderer has been further improved to these regards in 4.13. Changes affect the following low level rendering features.

Antialiasing

Lines now recognize the ILLines.Antialiasing property for thick lines (Width > 2). The antialised rendering works in all situations: transparent lines, lines with stipple patterns, inside /outside of plot cubes and for logartithmic / linear axis scales. The quality of the antialiasing implementation is now on par with common OpenGL implementations.

The new default value for ILLines.Antialiasing is now true. However, thin lines would not profit from antialiasing, hence thin lines will continue to ignore the value of the ILLines.Antialiasing property.

Note, that some OpenGL drivers actually refuse to render certain combinations of line properties. We experienced such behavior for stippled, thick line strips having antialiased rendering configured. In such situations you should make sure to have the most recent OpenGL / graphics card drivers installed. Alternatively you may chose either stippled pattern – / or antialiased rendering by explicitly configuring your lines for either one setting. Or use the GDI renderer instead.

Following is a screenshot of some lines examples. The left side shows the OpenGL renderer result. On the right side the GDI version is shown. Use your browser to show the full, unscaled image (right click on the image):

… and with dotted lines:

Default Color for ILLines & ILLineStrip

Basic, low level line shapes are now created with a default color: black. Higher level objects (as ILLinePlot, ILSplinePlot, Markers, ILSurface etc.) should always provide a color for low-level objects or use vertex based coloring explicitly. If you are assembling your scene with the low level line objects make sure to check that you have done this. This is not a breaking change.

Note that the setting for the shapes Color property overrides the vertex colors buffer (Colors property). In order to use vertex based coloring one must set the Color property to null, enabling the colors information from the Colors buffer.

Auto Color for Spline Plots

ILSplinePlot (derived from ILLinePlot) now adopts the auto-coloring feature from the ILLinePlot class. When no line color was given at the time the plot was created the color gets assigned which comes next in the ILLinePlot.NextColor color enumeration. Use the linecolor constructor argument of ILSplinePlot or the Line.Color property in order to control the color of the spline line explicitly.

Camera Default Depth: 100

In earlier versions the default ILCamera.ZFar value was 1000 which led to depth buffer precision issues in some situations. The new default value of 100 increases the depth buffer precision significantly. However, the new value (just like the old one) does not take into account the actual depth of your scene! If you encounter unwanted clipping in the far clippping area now, set the value for scene.Camera.ZFar explicitly. At best you know the depth of your scene and use this value. Or – more simple – use the old default of 1000:

 scene.Camera.ZFar = 1000; 

Visual Studio 2017 Compatibility

With the new version ILNumerics installs into the great, fresh new Visual Studio 2017 (released March 2017). … well, at least ‘kind of’…  What is true is that with 4.13 you can again use ILNumerics in all recent Visual Studio editions supporting extensions at all (which is all editions except Express Edition – I wonder if anyone is using this, still??), including Visual Studio 2017 Community Edition. This is great and all … but:

As you noticed there was a great deal of changes coming with VS2017. This includes the installer system which is – great attempt! – much more slim now by omitting unneeded stuff from the installation. But VS2017 also requires changes to the manifest files facilitating every Visual Studio Extensibility project. A new version has been introduced: version3. This is the first version which is not compatible with Visual Studio 2010 anymore. As a consequence, by supporting the new extension packaging system we would be required to drop support for Visual Studio 2010. This is not too dramatic since the 6 years VS2010 is out now feel kind of an eternity in our dev-world. However, we wanted to give the remaining VS2010 customers at least one version iteration of deprecating VS2010 in order to jump to a more recent version smoothly.

Additionally, the way the new VS installer works seems to reflect not the last word spoken on that topic (at least this is what we hope). The bottom line: we use an MSI installer, wrapped in an exe bootstrapper. The MSI installs all GAC binaries, registers the development dlls in Visual Studio, maintains the singleton installation directory and triggers the VSIX installer which installs the extension into all supported Visual Studio installations located on the system. This may sound complicated but worked quite reliably over the years.

Now, with the new Visual Studio package system things become odd. Out of subtle reasons, the MSI cannot (reliably) trigger the VSIX package automatically (and quietly). The reason becomes obvious when considering that the new extension potentially needs to trigger the installation of new components which it depends on. This install would not be possible while the hosting MSI is being installed itself still. So, as a consequence, currently, MSI installs of VSIX extensions into VS2017 are simply not working. When you start the ILNumerics installation from the delivered *.exe you will find the extension being installed into VS2010 and upwards – with the exception of Visual Studio 2017

Currently, some smart (WIX) people are attempting a solution to this. But we are not aware of a clean solution released already. Therefore, we will wait for it and/or eventually consider a new deployment scheme for our extensions.

However, luckily there is a simple workaround for now! After the ILNumerics installation was finished, you can easily install our extension into VS2017 manually. Just go to the installation folder (by default: C:\Program Files (x86)\ILNumerics\ILNumerics Ultimate VS\bin) and find the ‘ILNumerics.VSExtension.vsix’ file. Double click on it to start the installation into the remaining Visual Studio instances manually. This should work without problems. Be sure to accept the warning dialog during the install. It originates from the fact that our extension must still support older Visual Studio extension techniques.

Keep in mind that this is a workaround, though! Once installed the extension will work as expected. But some things will not work consistentyl. Deinstallation, for example must be done manually from the “Extensions and Tools” dialog within Visual Studio 2017:

Also, in difference to the machine wide installation done by the (administrative) MSI install the manual VSIX installation changes the local user account only. You may have to repeat the VSIX install for other user accounts. Besides these lowered installation experience we know of no other incompatibilities in Visual Studio 2017.

The post Release Notes ILNumerics 4.13 (detailed) appeared first on The ILNumerics Blog.

Broadcasting as a concept today is found in many popular mathematical prototyping systems. The most direct correspondence probably exists in the numpy package. Matlab and Octave offer similar functionality by means of the bsxfun function.

The term ‘broadcasting’ refers to a binary operator which is able to apply an elementwise operation to the elements of two n-dimensional arrays. The ‘operation’ is often as simple as a straight addition (ILMath.add, ILMath.divide, a.s.o.). However, what is special about broadcasting is that it allows the operation even for the case where both arrays involved do not have the same number of elements.

Broadcasting in ILNumerics prior Version 4.10

In ILNumerics, broadcasting is available for long already. But prior version 4.10 it was limited to scalars operating on n-dim arrays and vectors operating on matrices. Therefore, we had used the term ‘vector expansion’ instead of broadcasting. Obviously, broadcasting can be seen as a generalization of vector expansion.

Let’s visualize the concept by considering the following matrix A:

A

1  5   9  13  17
2  6  10  14  18
3  7  11  15  19
4  8  12  16  20

Matrix A might represent 5 data points of a 4 dimensional dataset as columns. One common requirement is to apply a certain operation to all datapoints in a similar way. In order to, let’s say, scale/weight the components of each dimension by a certain factor, one would multiply each datapoint with a vector of length 4.

ILArray<double> V = new[] { 0.5, 3.0, 0.5, 1.0 };

0.5
3.0
0.5
1.0

The traditional way of implementing this operation would be to expand the scaling vector by replicating it from a single column to a matrix matching the size of A.

VExp = repmat(V, 1, 5); 

0.5  0.5  0.5  0.5  0.5
3.0  3.0  3.0  3.0  3.0
0.5  0.5  0.5  0.5  0.5
1.0  1.0  1.0  1.0  1.0

Afterwards, the result can be operated with A elementwise in the common way.

ILArray<double> Result = VExp * A;

0.5   2.5   4.5   6.5   8.5
6.0  18.0  30.0  42.0  54.0
1.5   3.5   5.5   7.5   9.5
4.0   8.0  12.0  16.0  20.0

The problem with the above approach is that the vector data need to be expanded first. There is little advantage in doing so: a lot of new memory is being used up in order to store completely redundant data. We all know that memory is the biggest bottleneck today. We should prevent from lots of memory allocations whenever possible. This is where vector expansion comes into play. In ILNumerics, for long, one can prevent from the additional replication step and operate the vector on the matrix directly. Internally, the operation is implemented in a very efficient way, without replicating any data, without allocating new memory.

ILArray<double> Result = V * A;

0.5   2.5   4.5   6.5   8.5
6.0  18.0  30.0  42.0  54.0
1.5   3.5   5.5   7.5   9.5
4.0   8.0  12.0  16.0  20.0

Generalizing for n-Dimensions

Representing data as matrices is very popular in scientific computing. However, if the data are stored into arrays of other shapes, having more than two dimensions, one had to fall back to repmatting in order for the binary operation to succeed. This nuissance has been removed in version 4.10.

Now it is possible to apply broadcasting to two arrays of any matching shape – without the need for using repmat. In order for two arrays to ‘match‘ in the binary operation, the following rules must be fullfilled:

1. All corresponding dimensions of both arrays must match.
2. In order for two  corresponding dimensions to match,
   • both dimensions must be of the same length, or
   • one of the dimensions must be of length 1.

An example of two matching arrays would be a vector running along the 3rd dimension and a 3 dimensional array:

In the above image the vector (green) has the same length as the corresponding dimension of the 3D array (gray). The size of the vector is [1 x 1 x 6]. The size of the 3D array is [4 x 5 x 6]. Hence, any dimension of both, the vector and the 3D array ‘match’ in terms of broadcasting. A broadcasting operation for both, the vector and the array would give the same result as if the vector would be replicated along the 1st and the 2nd dimensions. The first element will serve all elements in the first 4 x 5 slice in the 1-2 plane. This slice is marked red in the next image: Note that all red elements here derive from the same value – from the first element of the green vector.  The same is true for all other vector elements: they fill corresponding slices on the 3D array along the 3rd dimension.

Slowly, a huge performance advantage of broadcasting becomes clear: the amount of memory saved explodes when more, longer dimensions are involved.

 Special Case: Broadcasting on Vectors

In the most general case and if broadcasting is blindly applied, the following special case potentially causes issues. Consider two vectors, one row vector and one column vector being provided as input parameters to a binary operation. In ILNumerics, every array carries at least two dimensions. A column vector of length 4 is considered an array of size [4 x 1]. A row vector of length 5 is considered an array of size [1 x 5]. In fact, any two vectors match according to the general  broadcasting rules.

As a consequence operating a row vector [1 x 5] with a column vector [4 x 1] results in a matrix [4 x 5]. The row vector is getting ‘replicated’ (again, without really executing the replication) four times along the 1st dimension, and the column vector 5 times along the rows.

array(new[] {1.0,2.0,3.0,4.0,5.0}, 1, 5) +array(new[] {1.0,2.0,3.0,4.0}, 4, 1)

<Double> [4,5]
[0]:          2          3          4          5          6
[1]:          3          4          5          6          7
[2]:          4          5          6          7          8
[3]:          5          6          7          8          9

Note, in order for the above code example to work, one needs to apply a certain switch:

Settings.BroadcastCompatibilityMode = false;

The reason is that in the traditional version of ILNumerics (just like in Matlab and Octave) the above code would simply not execute but throw an exception instead. Originally, binary operations on vectors would ignore the fact that vectors are matrices and only take the length of the vectors into account, operating on the corresponding elements if the length of both vectors do match. Now, in order to keep compatibility for existing applications, we kept the former behavior.

The new switch ‘Settings.BroadcastCompatibilityMode’ by default is set to ‘true’. This will cause the Computing Engine to throw an exception when two vectors of inequal length are provided to binary operators. Applying vectors of the same length (regardless of their orientation) will result in a vector of the same length.

If the ‘Settings.BroadcastCompatibilityMode’ switch is set to ‘false’ then general broadcasting is applied in all cases according to the above rules – even on vectors. For the earlier vector example this leads to the resulting matrix as shown above: operating a row on a column vector expands both vectors and gives a matrix of corresponding size.

Further reading: binary operators, online documentation

The post N-dim Array Broadcasting Efficiency in ILNumerics 4.10 appeared first on The ILNumerics Blog.

Problem during installing ILNumerics

Since the error is mostly related to the software upgrades, I was able to realize it on my computer. I have been playing with the installer and the build for quite some time now. I had an intermediate build being installed on my computer. I downloaded the official trial to use it for demonstration purpose. The following thing happened to me.

Without even changing a string, next, next… install, the progress bar is running. Suddenly the following message pops up:

Reasons why it might happen

Visual Studio potentially gets confused when trying to install / uninstall multiple installer versions. Especially when such installers have the same major and minor version number. If one receives a software upgrade, it usually means the major and minor version is not changed. Therefore it is not a considered as major or minor upgrade.

Locating the error

The easiest way to detect it, is after uninstalling ILNumerics. Check the VS Extensions (Tools -> Extensions and Updates). If you still see ‘ILNumerics Ultimate VS…’ installed that means you manually have to remove it. Unfortunately we have made some mistakes, and somehow it is not being removed correctly. This is something which happens occasionally.

Resolution

A suggestion is to remove the VS extension manually. It is not a normal, nor an intended way of handling (un)installing ILNumerics. It is a workaround. I did the following: from VS 2010 to 2013, I manually uninstalled the VS Extensions (note: VS 2010 needs to be started with admin rights to be able to remove the extension).

For me, after the removal of the extensions, the installer went smoothly Hope this helps in some unexpected cases! We would like to make the installation process smoother, feedback and ideas are welcome.

The post Installing ILNumerics – Unexpected behavior appeared first on The ILNumerics Blog.

During a lengthy beta phase we received a whole bunch of precise and enormous helpful feedback from you. We really appreciate that and again would like to say thanks!

ILNumerics Optimization Toolbox adds a number of functions to ILNumerics, useful to find solutions of common optimization problems. Since everything is nicely integrated into ILNumerics it helps you solve the problem easily and very efficiently. Optimization applications like the one shown as screenshot above can now get realized in a couple of minutes!

This blog post sheds some light on the very first steps for using the new Optimization Toolbox. It helps you to start quickly and lists some common documentation sources.

Obtaining the Optimization Toolbox

The optimization toolbox is available as a dedicated package and so must be individually purchased and installed. ILNumerics 4.6 or above is needed for that to work. Existing customers can evaluate the new toolbox by installing an extended trial on top of your existing ILNumerics Ultimate VS installation. Just let us know and we will lead you the way to the download. Another option is the trial: it includes all toolboxes, hence you can start right away by downloading and installing a trial of ILNumerics Ultimate VS and get familiar with all optimization methods easily.

Optimization Toolbox Setup

The optimization toolbox obviously depends on the ILNumerics Computation Engine. It installs the managed assemblies ILNumerics.Optimization.dll into the GAC and also makes them available inside Visual Studio as reference to your application projects:

Afterwards, the ILNumerics.Optimization class is found in the ILNumerics namespace which commonly is included in your source files anyway. A super-mini but complete starting example in a fresh new C# console application could look as follows:

using System;
using ILNumerics; 

namespace ConsoleApplication1 {
    class Program : ILMath {

        static ILRetArray<double> myObjFunc(ILInArray<double> A) {
            using (ILScope.Enter(A)) {
                return sum((A - 1) * A);
            }
        }
        static void Main(string[] args) {
            ILArray<double> M = Optimization.fmin(myObjFunc, ones(1,4));
            Console.WriteLine(M);
        }
    }
}

Note how we derived our console class from ILNumerics.ILMath! This allows us to omit the namespace.class identifier and ‘ILMath.sum’ and ‘ILMath.ones’ become just: ‘sum’ and ‘ones’.

If you are lucky enough to use Visual Studio 2015, it will allow you the inclusion of the static ILNumerics.Optimization class into your ‘using’ directives on top of your source code files also. This will inject all public functions of the Optimization class into your scope also which gives an even shorter syntax. ‘fmin’ and all other optimization functions are now readily available right next to the common ILMath functions:

That’s it already! Happy optimizing!

Optimization Examples

A number of example applications have been added to the examples section. Basically, they accompany the corresponding online documentation and let you follow the tutorials step by step on your own machine.

In order to start with an example application, you download the example as a zip package and extract it into a new folder on your local machine. Open the ‘*.csproj’ contained in the example package with Visual Studio. You will notice missing references in the References node of your project in the Solution Explorer:

Just remove these old references and replace them with references to the files actually installed on your system – using the common method described above.

Hit F5 and run the example!

Documentation

The ILNumerics Optimization Toolbox online documentation is available here:

http://ilnumerics.net/ilnumerics-optimization-toolbox.html

The API /class documentation for all functions of ILNumerics.Optimization is available here:

http://ilnumerics.net/apidoc/?topic=html/T_ILNumerics_Optimization.htm

Wrap up

We gave a short introduction in how to obtain and install ILNumerics Optimization Toolbox for ILNumerics Computing Engine. We pointed you to the common places of examples and documentation.

If you run into trouble or have useful suggestions and feedback, let us know! Contact sales@ilnumerics.net for any licensing questions.

The post Directions to the ILNumerics Optimization Toolbox appeared first on The ILNumerics Blog.

Recently, a user asked how to turn the background of a legend object in ILNumerics plots transparent. There doesn’t seem to be a straight forward way to that. One might expect code like the following to work:

var legend = new ILLegend("Line 1", "Line 2");
legend.Background.Color = Color.FromArgb(200, Color.White);

Unfortunately, it does not. Legend objects do not implement a ‘Background’ property which would give convenient access to the background object. However, the background object obviously exists. And ILLegend – likewise most of the more complex scene graph / plotting objects in ILNumerics – is derived from ILGroup! Group nodes represent the common way to store and manage any number of child objects. At the same time, group nodes gives the user flexible access to their children.
In case of ILLegend we can utilize one of the access methods of ILGroup in order to access the scene graph node which implements the background of the legend. Once we have the background object, we can go out and manipulate it freely. In order to do that, let’s first investigate, which nodes exist inside a legend object!

Stepping into living Scene Graphs

ILNumerics and Visual Studio make it very easy to investigate even complex scenes. Let’s create a simple line plot application first! We start with a fresh new C# Windows Forms Application and add a Plotting Form Template to it. A detailed instruction is given on the getting started page, at “Creating Visualizations with ILNumerics“.
Double click on the auto-generated Program.cs file and change the last line to read:

    Application.Run(new Plotting_Form1());

Starting the project via F5 brings up the newly generated example plot. We make one more addition. Double click the Plotting_Form1.cs to show the form in the designer. Now, double click on the white panel1 background to go to the code implementing the scene setup. At the end of the code creating the new plot cube and adding the line plot we add another line which adds a new legend object:

// Initial plot setup, modify this as needed
private void ilPanel1_Load(object sender, EventArgs e) {

    // create some test data, using our private computation module as inner class
    ILArray<float> Pos = Computation.CreateData(4, 300);

    // setup the plot (modify as needed)
    ilPanel1.Scene.Add(new ILPlotCube(twoDMode: false) {
        new ILLinePlot(Pos, tag: "mylineplot") {
            Line = {
                Width = 2,
                Color = Color.Red,
                Antialiasing = true,
                DashStyle = DashStyle.Dotted
            }
        }, new ILLegend("MyLineItem")
    });
    // register event handler for allowing updates on right mouse click:
    ilPanel1.Scene.First<ILLinePlot>().MouseClick += (_s, _a) => {
        if (_a.Button == MouseButtons.Right)
            Update(ILMath.rand(3, 30));
    };
}

Starting via F5 should give a form similar like this:
Now, let’s start the fun! Investigating living objects works best when the objects are alive, right? So let’s set a breakpoint in Visual Studio right after the legend has been created:

As you know, Visual Studio allows for a nice feature ‘Edit and Continue’. Let’s add a variable as reference to our legend right in Debug mode! This is not really necessary, but eases the inspection. Note, it does not matter, where to add the code line, as long as we step over it in the debugger:

    var leg = ilPanel1.Scene.First<ILLegend>();

Scene graph nodes are clever enough to give reasonable insights into their state and content at runtime. Just hover over the newly created variable ‘leg’ with the mouse and Visual Studio will display helpful debugger visualizers. Expand any node to show its content:

Inspecting the legend object that way makes it immediately clear, which object legends are made out of: a triangle shape realizes the background, a line strip shape the border. A common group holds the individual legend items and will be filled dynamically at rendering time.

Configuration Flexibility at its Extreme

Having access to every single child item in the scene graph obviously brings an extreme flexibility for configurations. Now it is easy to change the color of the background object to make the legend transparent:

var leg = ilPanel1.Scene.First<ILLegend>();
leg.First<ILTriangles>("ScreenObjectBackground").Color = Color.FromArgb(200, Color.White);

Which gives us our transparent legend:

Do not stop here!

This is the message to take away: any object in any visualization and any plot in ILNumerics is composed out of individual smaller objects. You can access them and manipulate them freely without limits! Most ‘convenience’ properties like “colorbar.Background” barely do more than we have done here: they give easy access to inner children (and add Intellisense support, of course). But if you need access – you can easily gain it yourself by inspecting the scene graph and its objects by the methods described here.

The post Getting to know your Scene Graph appeared first on The ILNumerics Blog.

In this blog post I am going to show how easy it is to extend the common functionality of Excel. We will enable an Excel Workbook to load arbitrary HDF5 data files, inspect the content of such files and show the data as interactive 2D or 3D plots. HDF5 is an industry standard for the structured storage of technical data and is maintained by the HDF Group. With version 4.0 ILNumerics supports the HDF5 file format with a very convenient, object oriented API. Visualizations have always been a popular feature since the early days of ILNumerics. And Excel can be another convenient GUI tool to marry both.

Prerequisites

Customizations like the one we are going to show are done in Visual Studio and utilize Visual Studio Tools for Office (VSTO). If you don’t own a copy of Visual Studio, you may find it useful that Microsoft gives away their flagship for free under certain conditions. Another prerequisite we will need: Office 2010 or 2013. I use Office 2010 on my computer. For some reason, Visual Studio 2013 did not allow me to create a new Workbook project for the 2010 version of Office and rather requested Office 2013. Therefore, I used Visual Studio 2012 instead. Just make sure to use the version of Office which is supported by your Visual Studio installation.

The last ingredient needed is ILNumerics. Download the free trial – it contains all features and tools you may possibly want and makes them available systemwide.

Setup

Let’s start with a fresh new Workbook project in Visual Studio:

We take all defaults here. This will create a new Visual Studio project with three worksheets, ready for your customizations. Double click on Sheet1.cs to open the designer view.  In the Visual Studio Toolbox find the ILNumerics section and drag a new instance of ILPanel onto your sheet:

This will add the reference of ILNumerics to your project and places the ILPanel control on your sheet. You may want to resize and reposition the control. It will be available from now on as ‘iLNumerics_Drawing_ILPanel1′ in your code-behind classes. Feel free to adjust the name – for this demonstration we will leave it as it is.

Loading HDF5 Files in Excel

Excel does not support HDF5 file imports directly (yet?). Luckily, ILNumerics bridges the gap very efficiently and conveniently. First, we add a reference to the HDF5 assembly which comes with ILNumerics. In the last step the reference to ILNumerics was added automagically. For HDF5 we have to do this manually. Right click on the References tab in the solution explorer and chose: Add Reference. Now search for “HDF5″ on your system, select the item found and hit “OK”:

If no item was found, make sure you have installed the latest ILNumerics package using our installer and that you selected the Assemblies tab in the Reference Manager window.

Once we have the HDF5 assembly available, we can start coding. The idea is, that the user should be able to load an HDF5 file from disk, inspect the datasets contained and get the option to load and/or visualize their data. So, let’s add some more controls from the toolbox to the workbook sheet: OpenFileDialog and a button to trigger the opening. Drag the button (Common Controls in the Toolbox) and the OpenFileDialog (Dialogs tab in the Toolbox) to the designer surface:

Now, rename the new button and double click it to open the auto-generated event handler method. This is where we are going to implement the code to open and inspect the HDF5 file:

private void button1_Click(object sender, EventArgs e) {
    var fod = new OpenFileDialog();
    if (fod.ShowDialog() == DialogResult.OK) {

        var filename = fod.FileName;
        // access the HDF5 file for reading
        using (var file = new H5File(filename, accessMode: FileAccess.Read)) {
            int row = 4; // start listing at row 4
            // list all datasets in the file
            foreach (var ds in file.Find<H5Dataset>()) {
                Cells[row,   3].Value = ds.Path;
                Cells[row++, 4].Value = ds.Size.ToString();
            }
            while (row < 100) {
                Cells[row, 3].Value = "";
                Cells[row++, 4].Value = "";
            }
            // display filename
            Cells[2, 4].Value = filename;
        }
    }
}

First, we ask for the filename to open. If the user provided a valid filename we open the file for reading. HDF5 files in ILNumerics are used in ‘using’ blocks. No matter how you leave the block, ILNumerics ensures that the file is not left open. Read here for more details.
Once we have the HDF5 file we start iterating over its datasets. C# foreach constructs make that really easy. Other languages have similar constructs. Inside the foreach loop we simply write out the path of the current dataset and its size to columns of the worksheet.
The while loop afterwards is only needed to clear entries potentially left over from earlier loadings. This expects no more than 100 datasets in a file. In a production code, you will do better…
Finally, the name of the HDF5 file is written into cell[2,4] for user information.

If we now run the workbook (hit F5) and click on our “Load HDF5 Dataset” button a file dialog opens up. Once we select an existing HDF5 file from our disk the file’s datasets are listed on the worksheet:

Loading HDF5 Dataset Content

Next, register a new event handler for the double click event in the worksheet. The worksheet template offers a good place to do so: the Sheet1_Startup event handler is auto generated by Visual Studio. Add the following line in order to allow to react to double click events on the sheet:

this.BeforeDoubleClick += Sheet1_BeforeDoubleClick;

The implementation of the Sheet1_BeforeDoubleClick method does all the work:

void Sheet1_BeforeDoubleClick(Excel.Range Target, ref bool Cancel) {
    // only take cells we are interested in
    if (Target.Value == null || Cells[2, 4].Value == null) return;
    // grab the hdf5 filename from the cell
    var curFilename = Cells[2, 4].Value.ToString();
    // check if this points to an existing file
    if (File.Exists(curFilename)) {
        // grab the dataset name (if the user clicked on it)
        var dsName = ((object)Target.Value).ToString();
        // reasonable?
        if (Target.Count == 1 && !String.IsNullOrEmpty(dsName)) {
            // open the file
            using (var file = new H5File(curFilename, accessMode: FileAccess.Read)) {
                // find the dataset in the file, we provide the full abs. path so we
                // are sure that there is only one such dataset
                var ds = file.First<H5Dataset>(dsName);
                if (ds != null) {
                    // add a new sheet with the name of the dataset
                    var sheet = (Excel.Worksheet)Globals.ThisWorkbook.Sheets.Add();
                    sheet.Name = checkName(dsName);
                    // ... and make it active
                    Globals.ThisWorkbook.Sheets[sheet.Name].Activate();
                    // load data using our extension method (see text)
                    sheet.Set(ds.Get<double>());
                } else {
                    // user has clicked on the size column -> plot the data
                    var size = ParseSize(dsName);
                    if (size != null && Target.Previous.Value != null) {
                        dsName = ((object)Target.Previous.Value).ToString();
                        // read data and render into panel
                        renderDS(file, dsName);
                    }
                }
            }
        }
    }
}

This is all straight forward: we do some very simple error checking here, just to make sure we only react to clicks on interesting columns. In your production code you will do much better error checking! However, here we decide if the user has clicked on a cell with a valid dataset name. If so, the file is opened (fetching the name from the HDF5 filename cell written to earlier), the dataset is located and its content loaded.
The content is written to a new workbook sheet. Attention must be drawn to the naming of the sheet. If a sheet with a similar name exists already, Excel will throw an exception. One easy solution is to add the timestamp to the name, which is left as an exercise. Here, we only do very simple name checking to make sure, no invalid characters enter the name of the sheeet:

string checkName(string name) {
    var ret = name.Replace('/','_');
    if (ret.Length > 31) {
        ret = ret.Substring(-31);
    }
    return ret;
}

The method to actually load the data from an ILArray to the new sheet is found in the following extension method. This is one rather efficient attempt to load large data:

public static void Set(this Worksheet worksheet, ILInArray<double> A, int fromColumn = 1, int fromRow = 1) {
    using (ILScope.Enter(A)) {

        var luCell = worksheet.Cells[fromRow, fromColumn];
        var rbCell = worksheet.Cells[fromRow + A.S[0] - 1, fromColumn + A.S[1] - 1];
        Range range = worksheet.Range[luCell, rbCell];
        range.Value = A.T.ToSystemMatrix();
    }
}

private static System.Array ToSystemMatrix<T>(this ILDenseArray<T> A) {
    using (ILScope.Enter(A)) {
        // some error checking (to be improved...)
        if (object.Equals(A, null)) throw new ArgumentException("A may not be null");
        if (!A.IsMatrix) throw new ArgumentException("Matrix expected");

        // create return array
        System.Array ret = Array.CreateInstance(typeof(T), A.S.ToIntArray().Reverse().ToArray());
        // fetch underlying system array
        T[] workArr = A.GetArrayForRead();
        // copy memory block
        Buffer.BlockCopy(workArr, 0, ret, 0, Marshal.SizeOf(typeof(T)) * A.S.NumberOfElements);
        return ret;
    }
}

Set() creates the range in the sheet to load the data into. The size is computed by the size of the incoming ILArray<T>. In order to load the data, we do not want to iterate over each
individual cell for performance reasons. One option is to set the Value property of the range to a two dimensional System.Array. ToSystemArray() does exactly that conversion. However, you have to
be careful not to get transposed data unexpectedly. The reason is that .NET multidimensional arrays are stored in row major order. ILNumerics on the other hand stores arrays in the same order as Matlab, FORTRAN and other technical tools do. Hence, we need to transpose our data before we assign them to the range. Read more details here.

Now, when we run the application and load a HDF5 file, we can double click on the cell holding a dataset name and have Excel load the dataset contents into a new worksheet – fast. This can be easily adopted by defining ranges (hyperslabs) and only load partial datasets. Also you can adopt the method described here for writing worksheet contents to HDF5 datasets.

Visualizing HDF5 Dataset Contents

Now let’s add another nice feature to our workbook: instead of simply loading the data from a dataset to the worksheet, we add the option of creating interactive, fully configurable and fast visualizations and plots of the data. We’ll use the predefined plot classes of ILNumerics Visualization Engine here.

Back to the double click event handler created earlier, we left out the path which is executed once the user clicked on the size displayed next to each dataset. What happens here is also straightforward.

First we parse the size to see, if it gives something reasonable (again, you’ll add better error checking for a production release). If so, we give the HDF5 file together with the dataset name to the renderDS() method which does the rendering:

private void renderDS(H5File file, string dsName) {
    var ds = file.First<H5Dataset>(dsName);
    if (ds != null) {
        using (ILScope.Enter()) {
            ILArray<float> A = ds.Get<float>();
            if (A.IsVector) {
                iLNumerics_Drawing_ILPanel1.Scene = new ILScene() {
                    new ILPlotCube(twoDMode: true) {
                        new ILLinePlot(A, markerStyle: MarkerStyle.Diamond, lineWidth: 2)
                    }
                };
            } else if (A.IsMatrix) {
                iLNumerics_Drawing_ILPanel1.Scene = new ILScene() {
                    new ILPlotCube(twoDMode: false) {
                        new ILSurface(A, colormap: Colormaps.Hot) {
                            UseLighting = true
                        }
                    }
                };
            }
            iLNumerics_Drawing_ILPanel1.Refresh();
        }
    }
}

This code does not need commenting. It fetches the dataset and loads its content into an ILArray<float>. A new scene replaces the existing one in the worksheet ILPanel. The new scene contains a plot cube and a line plot or a surface plot. Which one is created depends on the shape of the data. Vector sized data create a lineplot, matrices are rendered as a surface plot. In order to have the new scene show up, we must trigger a refresh on the panel.
Now, run the workbook, load a HDF5 file having some vector and/or matrix sized datasets, select a dataset by double clicking on its size cell. The plot is created according to the data. Like all visualizations in ILNumerics we can interact with the data: rotation/ zoom /pan is done with the left/right mouse buttons. And of course, you are free to apply any of the very flexible configuration options in order to customize your plots further. For line plots this includes: markers, dash styles, tex labeling, advanced coloring, logarithmic axes and much more …

Deploying ILNumerics Excel Workbooks

Once the workbook is ready for distribution, we build a Release version from Visual Studio and from now on – do not need Visual Studio anymore! In order to make any machine ready for handling HDF5 data and create Excel worksheets with nice visualizations on it, one only needs to run the ILNumerics installer and transfer the workbook to that machine. It will run on any platform, regardsless if 32 or 64 bit!

You can find the full example for downloading on our examples page.

The post Fun with HDF5, ILNumerics and Excel appeared first on The ILNumerics Blog.

Convenience and Speed

Another advantage is performance: by writing C = A + B, with A and B being large arrays, the inner implementation is able to choose the most efficient way of evaluating this expression. Here is, what ILNumerics internally does:

• Pointer artithmetics – we remove the obligatory bound checks on array accesses by using C# pointers.
• Cache aware implementations – the biggest bottleneck in modern processors is memory, as your know. ILNumerics implements all basic math functions in a way which allows to very efficiently profit from cache hierarchies in your processor. You will notice, if you time ILMath.sum over the rows of an array against a naive implementation.
• Loops over large arrays are unrolled, saving a great amount of bookkeeping overhead.
• Multithreading support – iterations over large arrays are split and distributed to multiple cores.

ILNumerics Accelerator

Here is, what we are planning to add to this list:

• Utilizing SIMD vector extensions (SSE, AVX). Today, this parallel potential is accessed via the (native) MKL in ILNumerics. There has been a recent update to MKL 11.2 in ILNumerics Ultimate VS 4.4 which brings AVX2 support on corresponding Intel processors. For the future, we are planning to bring in extendend support for all managed evaluations as well. This will be introduced with the release of our ILNumerics Accelerator. Stay tuned – this will be a very exciting release… !
• The Accelerator will also do onother highly efficient feature: removal of intermediate arrays. Expressions like C = A + 2 * B will no longer create an intermediate result for
  2 * B, but reformulate the expression before evaluation. Again: memory is the bottleneck and this will save a lot of memory transfer to the CPU.
• Having a vast amount of GPUs available in basically every desktop computer today makes one wanting to use that – especially when waiting for your results. ILNumerics Accelerator will do it! It will decide on its own during runtime, which part of your algorithms are best executed on which available hardware and do it in the most efficient way possible. Need a further speedup? Just buy a better graphics card!

Limitations of n-dim arrays

Nothing comes for free, right? All the above is great for large data on vectorized algorithms. But what if we have a tight loop over small ILArrays or an algorithm which basically does a vast amount of scalar computations?

The flexibility of ILArray certainly comes at a price here. It becomes clear if you imagine the difference between a single double value and its corresponding representation in terms of a scalar ILArray<double>. The former – while being a value type lives on the stack rather than on the managed heap. It is not involved in garbage collection and there is basically no cost involved for its construction, access and destruction – at least when compared to the full featured implementation of ILArray.

ILArray, on the other hand, is a class instance. It lives on the heap instead of the stack storage. Creating an instance corresponds to the creation with ‘new’. For the destruction ILNumerics is able to completely prevent the GC having to handle ILArray instances. But even this memory management comes with a cost, naturally.

So there are two sides of the medal: flexibility and better performance for ‘large’ data. Flexibility and slower speed for tiny … to scalar data. For most algorithms, having a reasonable amount of management of small data does not hurt performance. But if such operations are to be done in tight loops, one starts looking for alternatives.

Alternatives

Especially for small data the flexible subarray features of ILArray are often not needed anyways.  Such scalar algorithm parts are much more efficiently implemented by using data structures that are especially designed for it: System.ValueType and System.Array. The good news: ILNumerics makes it very easy to resort to such system types instead. Here comes a comprehensive list of options:

Scalar operations on an ILArray<T>

You may also consider the following references:
Stackoverflow posts dealing with the scalar access issue:
http://stackoverflow.com/questions/20944672/iteration-through-ilarraydouble-is-very-slow

http://stackoverflow.com/questions/19224626/performance-of-ilnumerics-of-simple-math-operations-over-vectors-vs-system-array

The post Performance on ILArray appeared first on The ILNumerics Blog.

If you need to convert from, let’s say ushort to float, you will not find any prepared conversion function in ILMath. Luckily, it is very easy to write your own:

Here comes a method which implements the conversion from ushort -> float. A straight forward version first:

        /// <summary>
        /// Convert ushort data to ILArray&lt;float>
        /// </summary>
        /// <param name="A">Input Array</param>
        /// <returns>Array of the same size as A, single precision float elements</returns>
        public static ILRetArray<float> UShort2Single(ILInArray<ushort> A) {
            using (ILScope.Enter(A)) {
                ILArray<float> ret = ILMath.zeros<float>(A.S);
                var retArr = ret.GetArrayForWrite();
                var AArr = A.GetArrayForRead();
                int c = 0;
                foreach (ushort a in A) {
                    retArr[c++] = a;
                }
                return ret;
            }
        }

This method is used like that:

            ushort[,] rawSensorData = new ushort[,] {{0,1,2},{3,4,5}};
            ILArray<float> converted = UShort2Single(rawSensorData);
            /*
             * <Single> [3,2]
             * [0]:          0          3
             * [1]:          1          4
             * [2]:          2          5
             */

            // continue working with 'converted' here...

The following method does the same but utilizes pointer arithmetic, hence it needs the /unsafe flag. Use this, if performance is critical and your data are sufficiently large:

        /// <summary>
        /// Convert ushort data to ILArray&lt;float> (unsafe version)
        /// </summary>
        /// <param name="A">Input Array</param>
        /// <returns>Array of the same size as A, single precision float elements</returns>
        public unsafe static ILRetArray<float> UShort2SingleUnsafe(ILInArray<ushort> A) {
            using (ILScope.Enter(A)) {
                ILArray<float> ret = ILMath.zeros<float>(A.S);
                var retArr = ret.GetArrayForWrite();
                var AArr = A.GetArrayForRead();

                fixed (ushort* pAArr = AArr)
                fixed (float* pRetArr = retArr) {
                    ushort* pInWalk = pAArr;
                    ushort* pInEnd = pAArr + A.S.NumberOfElements;
                    float* pRetWalk = pRetArr;
                    while (pInWalk < pInEnd) {
                        *(pRetWalk++) = /*implicit: (float)*/ (*(pInWalk++));
                    }
                }
                return ret;
            }
        }

The post Uncommon data conversion with ILArray appeared first on The ILNumerics Blog.

* Evaluate options to use ILNumerics from IronPython

Several years ago there has been some attempts from ILNumerics users who successfully utilized ILNumerics from within IronPython. But despite our fascination for these attempts, we were not able to catch up and deeply evaluate all options for joining both projects. Years went by and Microsoft has dropped support for IronPython in the meantime. Nevertheless, a considerably large community seems to be active on IronPython. Finally, today is the day I am going to give this a first quick shot.

Disclaimer: I am not a python developer. My experience with IronPython range from 0 … zero (unfortunatley, since the project deserves much more attention). For CPython it is only slighly better. So please bare with me if I went into some stupid direction or if I have completely missed better options in this article! Corrections and suggestions are always warmly welcome.

Setup

I downloaded and installed IronPython from CodePlex. Also, in Visual Studio 2013 there exists a link in the NEW PROJECT dialog advertising the download of IronPython Tools for Visual Studio. I used that to setup Visual Studio for IronPython projects. All setup went smooth and easy. Nice!

Creating Plots with IronPython

The challenge I was interested in the most was how it is possible to utilize the great plotting capabilities of ILNumerics Visualization Engine from IronPython projects. Since matplotlib seems not to be available for IronPython and other alternatives are also pretty rare (if any at all?) having our visualization engine available to IronPython projects seem to be a big improvement.

The good news first: it works and it does so very easily. The following plot is done purely in IronPython:

I started with a fresh new Iron Python Windows Forms Application from the ‘New Project’ dialog in Visual Studio 2013.

This gives a template python file ‘IronPython_ILPanel.py’ with the following content:

import clr
clr.AddReference('System.Drawing')
clr.AddReference('System.Windows.Forms')

from System.Drawing import *
from System.Windows.Forms import *

class MyForm(Form):
    def __init__(self):
        # Create child controls and initialize form
        pass

Application.EnableVisualStyles()
Application.SetCompatibleTextRenderingDefault(False)

form = MyForm()
Application.Run(form)

If you are familiar with the common setup of Windows Forms you may notice some similarities with the stub provided by a new C# Windows Forms Application project. What is otherwise spread over multiple files, now is done all within the global scope of the one and only *.py file: the form class is defined, the static Application class is configured and the event loop is started with an instance of the formerly defined MyForm class. Straight. This corresponds to the common setup for Windows.Forms from any language, where no Designer support exists for. The same could be done in PowerShell, to name just another example.

In order to integrate support for ILNumerics Visualization Engine, I did the following simple steps:

1) If not done so far, go and install ILNumerics Ultimate VS. We offer 30 days trials which contain all features without limitation.
2) Once installed, ILNumerics is available in the “Add Reference” dialog (right click on References in the Solution Explorer). ILNumerics.dll is added from the .NET tab:

3) Make the ILNumerics namespaces available: I added the following lines to the import section at the very top of IronPython_ILPanel.py:

import clr
clr.AddReference('System.Drawing')
clr.AddReference('System.Windows.Forms')
clr.AddReference('ILNumerics')

from System.Drawing import *
from System.Windows.Forms import *
from ILNumerics import *
from ILNumerics.Drawing import *
from ILNumerics.Drawing.Plotting import *

4) Now we are ready to add a plot to the form. In C# we would simply use the ilPanel1_Load event handler which is easily added by help of the forms designer. Since we do not have designer support here, we simply add the setup to the constructor of MyForm:

class MyForm(Form):
   def __init__(self):
     # Create child controls and initialize form
     ilpanel = ILPanel()
     ilpanel.Dock = DockStyle.Fill
     self.Controls.Add(ilpanel)
     # show some demo: first create a plotcube
     pc = ILPlotCube("pc", 0)
     # it will hold a surface with some sinc data in 3D
     # You can use ILRetArray returned from any ILNumerics Computing
     # Module as input argument here directly. Type conversions seem to happen automatically.
     sf = ILSurface(ILSpecialData.sincf(40,50))
     pc.Add(sf)
     # add the plotcube to the scene
     ilpanel.Scene.Add(pc)
     self.Text = "Plotting Fun with ILNumerics in IronPython"

This is all straightforward. The configuration of the panel and the plots is exactly as it would have been done in C# or Visual Basic. However, we have to add the ILPanel to the self.Controls collection manually. The result, of course is exactly the same: a nice interactive form, utilizing OpenGL, with all manipulation options like rotation, pan and zoom. The full spectrum of the Visualization Engine should work out of the box. This includes several, flexible 3D and 2D plotting types as well as the whole scene graph functionality for building your own interactive visualizations.

This is all what needs to be done in order to create nice professional plots directly within IronPython. Press F5 and start your application, rotate the plot with the mouse and learn about all the options you have.

Computing Fun with IronPython and ILNumerics

ILNumerics not only rely on its own flexible n-dimensional array implementation, it also introduces its very own memory management – for very good reasons. In order for the memory managements to work, individual array types (ILArray, ILInArray, ILOutArray and ILRetArray) are used in ILNumerics algorithms. This makes great use of the strict type safety of .NET languages as C# and Visual Basic and of automatic conversions between those types.

Python on the other side is a dynamic language. It does not know the concept of types to the same extend. A straightforward application of the ILNumerics array types is not possible. IronPython, however offers workarounds (clr.Convert) but they are not able to provide the same syntactical convenience as a pure C# algorithm.

My recommendation for the utilization of ILNumerics Computing Engine therefore is as follows:

ILNumerics Computing Engine can be used without restriction. The utilization of existing algorithms is straightforward. Algorithms leveraging the ILNumerics Function Rules can be called directly without any type conversions. The creation of local variables on the other side requires type conversions from ILRetArray to ILArray. This can be done by help of IronPythons clr.Convert function.

The type conversion issue possibly makes it less feasible to write own ILNumerics Computing Engine algorithms in IronPython. But most the time, one would rather want to use existing python algorithms anyway. In order to actually create a new algorithm, one should rather utilize C# and compile the algorithm into its own .NET module which can than easily be imported into your python project and get interfaced from your python code.

Summary

This blog demonstrated how easy it is to utilize ILNumerics from IronPython. Especially the Visualization Engine is incorporated without any problems and offers the full set of visualization and plotting options out of the box. Algorithms created with ILNumerics Computation Engine can directly be interfaced and used – in parallel with numpy algorithms, if the need arise. The syntax of the Computing Engine however is not as expressive and more clumsy due to the absense of implicit type conversions and operator overloads. Complex algorithms one therefore would rather implement in C# or VisualBasic than in IronPython. For small algorithms, like data preprocessing and the like, ILNumerics Computing Engine serves as a faster alternative to numpy arrays.

The post Plotting Fun with ILNumerics and IronPython appeared first on The ILNumerics Blog.

And here comes the code used to create this example:

private void ilPanel1_Load(object sender, EventArgs e) {
    // create some test data
    ILArray<float> A = ILSpecialData.torus(1.3f, 0.6f);

    // create the plot: a simple surface
    ilPanel1.Scene.Add(new ILPlotCube(twoDMode: false) {
        new ILSurface(A, colormap: Colormaps.Summer) {
            // we also want a colorbar
            new ILColorbar() {
                Background = {
                    Color = Color.DarkGray
                }
            }
        }
    });

    // set the backcolor of the scene to black
    ilPanel1.BackColor = Color.Black; 

    // set labels color
    foreach (var label in ilPanel1.Scene.Find<ILLabel>()) {
        label.Color = Color.White;
        label.Fringe.Width = 0;
    }

    // set the color of the default labels for axis ticks
    foreach (var axis in ilPanel1.Scene.Find<ILAxis>()) {
        axis.Ticks.DefaultLabel.Color = Color.White;
        axis.Ticks.DefaultLabel.Fringe.Width = 0;
    }

    // some more configuration: the view limits
    ilPanel1.Scene.First<ILPlotCube>().Limits.Set(
        new Vector3(0, 0, 1), new Vector3(2, 2, -1));

}

In line 4 we use the ILSpecialData class to create some test data. torus() creates the X, Y and Z values which eventually assemble a torus when used in ILSurface. The next line creates and adds a new plot cube to the scene. We set its two2Mode property to false, so we can rotate the torus with the mouse.

The next line creates a new surface and provides the torus data to it. As colormap ‘Colormaps.Summer’ is configured. Most surfaces need a colorbar in order to help mapping colors to actual values. We add a new colorbar below the surface and set its background color to some dark value.

Next, the BackColor of the main panel is set to black. Note, that setting the background color of a panel must be done in code in the current version (3.3.3). This is due to a bug in ILPanel which causes settings made in the designer to be ignored!

Now we have a dark background color but the labels still remain black. So let’s fix this: all labels which are part of the regular scene graph can easily be set at once. We simply use the ILGroup.Find() function to enumerate all labels and set their color to white. Also, we remove the fringe around them. Alternatively we could have set the fringe color to some dark color.

The last issue remaining is caused by the fact that labels for ticks cannot be configured here. The reason is, that tick labels are created dynamically. they don’t even exist at the time of execution of this code. So we must configure a thing called ‘DefaultLabel‘ instead. DefaultLabel is a member of the ticks collection of every axis object and used at runtime to provide default properties for all tick labels in auto mode.

This gives a nice dark color scheme. Keep in mind that the default color values for all scene-/plot objects are currently optimized for light background colors. Using dark backgrounds, therefore requires one to adjust the color on all plot objects accordingly.

The post Dark color schemes with ILPanel appeared first on The ILNumerics Blog.