HDF5 is a file format (Hierarchical Data Format) especially desgined to handle huge amount of numerical data. Just to mention an example,  NASA chose it to be the standard file format for storing data from the Earth Observing System (EOS).

ILNumerics easily handles HDF5 files. They can be used to exchange data with other software tools, for example Matlab mat files. In this post I will show a step by step guide – how to interface ILNumerics with Matlab.

Save Matlab data to HDF5 format

Let’s start by creating some arbitrary data with Matlab and saving it in HDF5 format, Matlab supports it from 2006 September, version 7.3.

matrix = randn(10, 10) * 5;
vector = [1 2 3 4 5]';
carr = 'It is a char array!';
noise = rand(1,100) - rand(1, 100);
sinus = 10 * sin(linspace(0, 2*pi, 100)) + noise;
complex = rand(1, 5) * (10 + 10i);
save('test_hdf5.mat', '-v7.3');

Load Matlab data with ILNumerics

Let’s now see what ILNumerics sees

Matlab simply stored all arrays as individual datasets in the root node of the HDF5 file. It is now easily possible to access the data! We could take a peek and inspect them right within the Visual Studio debugger visualizers. But let’s see how the data is loaded programmatically:

// Reading from Matlab
using (H5File file = new H5File("test_hdf5.mat"))
{
    using (ILScope.Enter())
    {
        ILArray<double> matrix = file.Get<H5Dataset>("matrix").Get<double>();
        ILArray<double> vector = file.Get<H5Dataset>("vector").Get<double>();
        ILArray<char> carr = file.Get<H5Dataset>("carr").Get<char>();
        ILArray<double> noise = file.Get<H5Dataset>("noise").Get<double>();
        ILArray<double> sinus = file.Get<H5Dataset>("sinus").Get<double>();
    }
}

The datasets are loaded – and the whole process was very easy. It took me less than ten minutes to figure out the methods. It would have been even less if I had take a look on the examples of ILNumerics

Now, let’s startup the Array Visualizer! Just write ‘array’ in the Quick Launch box of Visual Studio, or open a new visualizer window via VIEW -> OTHER WINDOWS -> Array Visualizer. In the visualizer expression field enter: ‘sinus’. The array is now visualized, how cool is that

Visualizing Matlab mat files – on the fly

Now we have shown the content of the variable ‘sinus’. Just as well, we could visualize the content of the HDF5 file and have the visualizer follow any changes to it immediately:

If we multiply it with -1, the expression is immediately evaluated:

And what is even more interesting, if we change the HDF5 file interactively, in the immediate window, the ILNumerics Array Visualizer reflects that on the fly:

Loading ILArrays into Matlab

Now let us check, what happens if we want to go from ILNumerics to Matlab. Saving a noise cosine with the following code:

// Writing some data to HDF5
// Note: It will not make it a proper mat file, only HDF5!
if (File.Exists("test_iln.mat"))
    File.Delete("test_iln.mat");

using (H5File newFile = new H5File("test_iln.mat"))
{
    using (ILScope.Enter())
    {
        ILArray<double> noisyCos = ILMath.cos(ILMath.linspace(0, 3 * Math.PI, 1000)) * 10 + ILMath.rand(1, 1000);
        H5Dataset set = new H5Dataset("noisyCos", noisyCos);
        newFile.Add(set);
    }
}

Since the ILNumerics HDF5 is not a mat file format, the only way to load it (as for now) is with one of the ‘h5*’ methods of Matlab. To check the info and load, the following code is used:

h5info('test_iln.mat', '/')
noisyCos = h5read('test_iln.mat', '/noisyCos');

In this way, it is possible to load ILArrays into Matlab

Altering Matlab mat files in ILNumerics

It is possible to alter loaded mat files with some limitations. What I found is that the length of the array can not be changed. Obviously, Matlab mat files are created as datasets without chunks, so the dataspace size cannot be changed after the file was created. However, one can alter the data (without changing the size) :

// Altering data of mat file (v7.3 format)
using (H5File file = new H5File("test_hdf5.mat"))
{
    using (ILScope.Enter())
    {
        ILArray<double> sinus = file.Get<H5Dataset>("sinus").Get<double>();
        sinus.a = sinus * -1;
        file.Get<H5Dataset>("sinus").Set(sinus, "0;:");
    }
}

Let me draw your attention to the use of the Set function here. I first tried to use:

file.Get<H5Dataset>("dsname").Set(A);

But this would attempt to overwrite the whole dataset – something which could potentially change the size, hence would require a chunked dataset. Therefore, I figured that I have to be more specific and need to provide the target data range also:

file.Get<H5Dataset>("sinus").Set(sinus, "0;:")

More Limitations

Out of curiosity I checked the complex array, to see if it is okay.

Unfortunately it throws a H5Exception, this is something I will look into and work on. Fixing this problem will require support for compound datatypes in our HDF5 API.

Try out the example

To play with it, head to the examples and download the code:
Reading and Writing Matlab data with ILNumerics

Conclusion

To sum up, it is possible and easy to load data stored in Matlab mat files to ILNumerics with the HDF5 IO handling, and it is also possible to load ILArrays to Matlab as well.

I am opened to suggestions and feedback, looking forward to hearing from you! Share your experiences with us, so we can learn from you!

The post HDF5 and Matlab Files – Fun with ILNumerics appeared first on The ILNumerics Blog.

In this blog post I am going to show how easy it is to extend the common functionality of Excel. We will enable an Excel Workbook to load arbitrary HDF5 data files, inspect the content of such files and show the data as interactive 2D or 3D plots. HDF5 is an industry standard for the structured storage of technical data and is maintained by the HDF Group. With version 4.0 ILNumerics supports the HDF5 file format with a very convenient, object oriented API. Visualizations have always been a popular feature since the early days of ILNumerics. And Excel can be another convenient GUI tool to marry both.

Prerequisites

Customizations like the one we are going to show are done in Visual Studio and utilize Visual Studio Tools for Office (VSTO). If you don’t own a copy of Visual Studio, you may find it useful that Microsoft gives away their flagship for free under certain conditions. Another prerequisite we will need: Office 2010 or 2013. I use Office 2010 on my computer. For some reason, Visual Studio 2013 did not allow me to create a new Workbook project for the 2010 version of Office and rather requested Office 2013. Therefore, I used Visual Studio 2012 instead. Just make sure to use the version of Office which is supported by your Visual Studio installation.

The last ingredient needed is ILNumerics. Download the free trial – it contains all features and tools you may possibly want and makes them available systemwide.

Setup

Let’s start with a fresh new Workbook project in Visual Studio:

We take all defaults here. This will create a new Visual Studio project with three worksheets, ready for your customizations. Double click on Sheet1.cs to open the designer view.  In the Visual Studio Toolbox find the ILNumerics section and drag a new instance of ILPanel onto your sheet:

This will add the reference of ILNumerics to your project and places the ILPanel control on your sheet. You may want to resize and reposition the control. It will be available from now on as ‘iLNumerics_Drawing_ILPanel1′ in your code-behind classes. Feel free to adjust the name – for this demonstration we will leave it as it is.

Loading HDF5 Files in Excel

Excel does not support HDF5 file imports directly (yet?). Luckily, ILNumerics bridges the gap very efficiently and conveniently. First, we add a reference to the HDF5 assembly which comes with ILNumerics. In the last step the reference to ILNumerics was added automagically. For HDF5 we have to do this manually. Right click on the References tab in the solution explorer and chose: Add Reference. Now search for “HDF5″ on your system, select the item found and hit “OK”:

If no item was found, make sure you have installed the latest ILNumerics package using our installer and that you selected the Assemblies tab in the Reference Manager window.

Once we have the HDF5 assembly available, we can start coding. The idea is, that the user should be able to load an HDF5 file from disk, inspect the datasets contained and get the option to load and/or visualize their data. So, let’s add some more controls from the toolbox to the workbook sheet: OpenFileDialog and a button to trigger the opening. Drag the button (Common Controls in the Toolbox) and the OpenFileDialog (Dialogs tab in the Toolbox) to the designer surface:

Now, rename the new button and double click it to open the auto-generated event handler method. This is where we are going to implement the code to open and inspect the HDF5 file:

private void button1_Click(object sender, EventArgs e) {
    var fod = new OpenFileDialog();
    if (fod.ShowDialog() == DialogResult.OK) {

        var filename = fod.FileName;
        // access the HDF5 file for reading
        using (var file = new H5File(filename, accessMode: FileAccess.Read)) {
            int row = 4; // start listing at row 4
            // list all datasets in the file
            foreach (var ds in file.Find<H5Dataset>()) {
                Cells[row,   3].Value = ds.Path;
                Cells[row++, 4].Value = ds.Size.ToString();
            }
            while (row < 100) {
                Cells[row, 3].Value = "";
                Cells[row++, 4].Value = "";
            }
            // display filename
            Cells[2, 4].Value = filename;
        }
    }
}

First, we ask for the filename to open. If the user provided a valid filename we open the file for reading. HDF5 files in ILNumerics are used in ‘using’ blocks. No matter how you leave the block, ILNumerics ensures that the file is not left open. Read here for more details.
Once we have the HDF5 file we start iterating over its datasets. C# foreach constructs make that really easy. Other languages have similar constructs. Inside the foreach loop we simply write out the path of the current dataset and its size to columns of the worksheet.
The while loop afterwards is only needed to clear entries potentially left over from earlier loadings. This expects no more than 100 datasets in a file. In a production code, you will do better…
Finally, the name of the HDF5 file is written into cell[2,4] for user information.

If we now run the workbook (hit F5) and click on our “Load HDF5 Dataset” button a file dialog opens up. Once we select an existing HDF5 file from our disk the file’s datasets are listed on the worksheet:

Loading HDF5 Dataset Content

Next, register a new event handler for the double click event in the worksheet. The worksheet template offers a good place to do so: the Sheet1_Startup event handler is auto generated by Visual Studio. Add the following line in order to allow to react to double click events on the sheet:

this.BeforeDoubleClick += Sheet1_BeforeDoubleClick;

The implementation of the Sheet1_BeforeDoubleClick method does all the work:

void Sheet1_BeforeDoubleClick(Excel.Range Target, ref bool Cancel) {
    // only take cells we are interested in
    if (Target.Value == null || Cells[2, 4].Value == null) return;
    // grab the hdf5 filename from the cell
    var curFilename = Cells[2, 4].Value.ToString();
    // check if this points to an existing file
    if (File.Exists(curFilename)) {
        // grab the dataset name (if the user clicked on it)
        var dsName = ((object)Target.Value).ToString();
        // reasonable?
        if (Target.Count == 1 && !String.IsNullOrEmpty(dsName)) {
            // open the file
            using (var file = new H5File(curFilename, accessMode: FileAccess.Read)) {
                // find the dataset in the file, we provide the full abs. path so we
                // are sure that there is only one such dataset
                var ds = file.First<H5Dataset>(dsName);
                if (ds != null) {
                    // add a new sheet with the name of the dataset
                    var sheet = (Excel.Worksheet)Globals.ThisWorkbook.Sheets.Add();
                    sheet.Name = checkName(dsName);
                    // ... and make it active
                    Globals.ThisWorkbook.Sheets[sheet.Name].Activate();
                    // load data using our extension method (see text)
                    sheet.Set(ds.Get<double>());
                } else {
                    // user has clicked on the size column -> plot the data
                    var size = ParseSize(dsName);
                    if (size != null && Target.Previous.Value != null) {
                        dsName = ((object)Target.Previous.Value).ToString();
                        // read data and render into panel
                        renderDS(file, dsName);
                    }
                }
            }
        }
    }
}

This is all straight forward: we do some very simple error checking here, just to make sure we only react to clicks on interesting columns. In your production code you will do much better error checking! However, here we decide if the user has clicked on a cell with a valid dataset name. If so, the file is opened (fetching the name from the HDF5 filename cell written to earlier), the dataset is located and its content loaded.
The content is written to a new workbook sheet. Attention must be drawn to the naming of the sheet. If a sheet with a similar name exists already, Excel will throw an exception. One easy solution is to add the timestamp to the name, which is left as an exercise. Here, we only do very simple name checking to make sure, no invalid characters enter the name of the sheeet:

string checkName(string name) {
    var ret = name.Replace('/','_');
    if (ret.Length > 31) {
        ret = ret.Substring(-31);
    }
    return ret;
}

The method to actually load the data from an ILArray to the new sheet is found in the following extension method. This is one rather efficient attempt to load large data:

public static void Set(this Worksheet worksheet, ILInArray<double> A, int fromColumn = 1, int fromRow = 1) {
    using (ILScope.Enter(A)) {

        var luCell = worksheet.Cells[fromRow, fromColumn];
        var rbCell = worksheet.Cells[fromRow + A.S[0] - 1, fromColumn + A.S[1] - 1];
        Range range = worksheet.Range[luCell, rbCell];
        range.Value = A.T.ToSystemMatrix();
    }
}

private static System.Array ToSystemMatrix<T>(this ILDenseArray<T> A) {
    using (ILScope.Enter(A)) {
        // some error checking (to be improved...)
        if (object.Equals(A, null)) throw new ArgumentException("A may not be null");
        if (!A.IsMatrix) throw new ArgumentException("Matrix expected");

        // create return array
        System.Array ret = Array.CreateInstance(typeof(T), A.S.ToIntArray().Reverse().ToArray());
        // fetch underlying system array
        T[] workArr = A.GetArrayForRead();
        // copy memory block
        Buffer.BlockCopy(workArr, 0, ret, 0, Marshal.SizeOf(typeof(T)) * A.S.NumberOfElements);
        return ret;
    }
}

Set() creates the range in the sheet to load the data into. The size is computed by the size of the incoming ILArray<T>. In order to load the data, we do not want to iterate over each
individual cell for performance reasons. One option is to set the Value property of the range to a two dimensional System.Array. ToSystemArray() does exactly that conversion. However, you have to
be careful not to get transposed data unexpectedly. The reason is that .NET multidimensional arrays are stored in row major order. ILNumerics on the other hand stores arrays in the same order as Matlab, FORTRAN and other technical tools do. Hence, we need to transpose our data before we assign them to the range. Read more details here.

Now, when we run the application and load a HDF5 file, we can double click on the cell holding a dataset name and have Excel load the dataset contents into a new worksheet – fast. This can be easily adopted by defining ranges (hyperslabs) and only load partial datasets. Also you can adopt the method described here for writing worksheet contents to HDF5 datasets.

Visualizing HDF5 Dataset Contents

Now let’s add another nice feature to our workbook: instead of simply loading the data from a dataset to the worksheet, we add the option of creating interactive, fully configurable and fast visualizations and plots of the data. We’ll use the predefined plot classes of ILNumerics Visualization Engine here.

Back to the double click event handler created earlier, we left out the path which is executed once the user clicked on the size displayed next to each dataset. What happens here is also straightforward.

First we parse the size to see, if it gives something reasonable (again, you’ll add better error checking for a production release). If so, we give the HDF5 file together with the dataset name to the renderDS() method which does the rendering:

private void renderDS(H5File file, string dsName) {
    var ds = file.First<H5Dataset>(dsName);
    if (ds != null) {
        using (ILScope.Enter()) {
            ILArray<float> A = ds.Get<float>();
            if (A.IsVector) {
                iLNumerics_Drawing_ILPanel1.Scene = new ILScene() {
                    new ILPlotCube(twoDMode: true) {
                        new ILLinePlot(A, markerStyle: MarkerStyle.Diamond, lineWidth: 2)
                    }
                };
            } else if (A.IsMatrix) {
                iLNumerics_Drawing_ILPanel1.Scene = new ILScene() {
                    new ILPlotCube(twoDMode: false) {
                        new ILSurface(A, colormap: Colormaps.Hot) {
                            UseLighting = true
                        }
                    }
                };
            }
            iLNumerics_Drawing_ILPanel1.Refresh();
        }
    }
}

This code does not need commenting. It fetches the dataset and loads its content into an ILArray<float>. A new scene replaces the existing one in the worksheet ILPanel. The new scene contains a plot cube and a line plot or a surface plot. Which one is created depends on the shape of the data. Vector sized data create a lineplot, matrices are rendered as a surface plot. In order to have the new scene show up, we must trigger a refresh on the panel.
Now, run the workbook, load a HDF5 file having some vector and/or matrix sized datasets, select a dataset by double clicking on its size cell. The plot is created according to the data. Like all visualizations in ILNumerics we can interact with the data: rotation/ zoom /pan is done with the left/right mouse buttons. And of course, you are free to apply any of the very flexible configuration options in order to customize your plots further. For line plots this includes: markers, dash styles, tex labeling, advanced coloring, logarithmic axes and much more …

Deploying ILNumerics Excel Workbooks

Once the workbook is ready for distribution, we build a Release version from Visual Studio and from now on – do not need Visual Studio anymore! In order to make any machine ready for handling HDF5 data and create Excel worksheets with nice visualizations on it, one only needs to run the ILNumerics installer and transfer the workbook to that machine. It will run on any platform, regardsless if 32 or 64 bit!

You can find the full example for downloading on our examples page.

The post Fun with HDF5, ILNumerics and Excel appeared first on The ILNumerics Blog.

Lets say, you went to that conference and want to give your new friends some insight into your brand new simulation results. The PC in the internet cafe enables you to fetch the data from your NAT storage at home. But will you be able to do anything with it on that plain Windows PC?

Or you want to localize a certain test data set but cannot remember its rather cryptic name. Or you might want to manage the latest measurement results from todays atmospheric observation satellite scans. The data are huge but often require some sort of preprocessing. There should be some easy way to filter them by the meta data within the files, right?

Other than getting the data from some application layer, we now want to interface plain old file objects. Of course, you store your data in HDF5 format, right? You do so, because HDF5 is portable, very efficient, flexible and you are in good company.

Let’s see. We have a fresh Windows PC and we know every Windows installation nowadays comes with Powershell. Powershell itself is based on the .NET framework and hence efficiently handles any .NET assembly. It should be easy to use ILNumerics with Powershell! All we still need is some way to access the HDF5 files. ILNumerics, natively is able to read and write Matlab mat files up to version 6. It currently lags on native HDF5 support.

Luckily, the HDF Group provides a large collection of high quality tools for HDF support. Among them you’ll find a .NET wrapper and … a brand new Powershell module: PSH5X! Together with Gerd Heber, the leading inventor of PSH5X, we did a feasibility study with the goal to investigate the options of utilizing HDF5 and ILNumerics together in Powershell. It can be downloaded here. We were quite impressed by the options this brings.

This blog post will describe the necessary steps to setup Powershell for ILNumerics and HDF5.

Getting Started

Basically, the installation process for any Powershell module consists of

1. Getting the module files and its dependencies from somewhere,
2. Deploying the module files into a special folder on your machine, and
3. Importing the module in your session.

The PSH5X homepage gives all information on how to get ready using the HDF5 Powershell module. Just download the package and follow the three steps on the page. At the end, HDF5 signals you a successful installation by displaying its version numbers.

Since ILNumerics depends on several other modules, we provide a small bootstrapper script. Just open up your favorite Powershell IDE (PowerShell_ISE.exe comes with any recent Windows) and copy/paste the following line:

(new-object Net.WebClient).DownloadString('http://ilnumerics.net/media/InstallILNumericsPSM.ps1') | iex

If you are curious, what this does – just ommit the trailing | iex and the script is not executed but displayed for your inspection.

The installer will ask for the installation folder (global under System32/ or local in your user profile), fetches the latest ILNumerics package for the current platform from the official nuget repository and install it into the selected module folder. In addition it loads the TypeAccelerator Powershell module and installs it into the same module directory. Note, the accelerators have been slightly modified in order to make them work with Powershell 3 and hence are fetched from our ILNumerics server. However, credits fully belong to poshoholic for his great work.

Note, the installation has to be done only once. Afterwards, on the next Powershell session, simply re-import needed modules by typing – lets say:

PS> Import-Module ILNumerics 

Go!

If everything was setup correctly, we can now use the full spectrum of the involved modules:

PS> [ilmath]::rand(4,5).ToString()
<Double> [5,4]
   0,72918    0,87547    0,43167    0,94942
   0,58024    0,75562    0,96125    0,83148
   0,22454    0,20583    0,82285    0,83144
   0,13300    0,40047    0,58829    0,87012
   0,50751    0,05496    0,02814    0,48764 

Nice. But what about the MKL? Are the correct binaries really installed as well?

PS> [ilf64] $A = [ilmath]::rand(1000,1000)
PS> Measure-Command { [ilf64]$C = [ilmath]::rank($A) }
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 920
Ticks             : 9202311
TotalDays         : 1,06508229166667E-05
TotalHours        : 0,00025561975
TotalMinutes      : 0,015337185
TotalSeconds      : 0,9202311
TotalMilliseconds : 920,2311

PS> $C.ToString()
1000

We have almost all options from C#:

PS> [ilf64] $part = $A['10:15;993:end']
PS> $part.ToString()
<Double> [11,7]
   0,08522    0,87217    0,59997    0,57363    0,22956    0,02006    0,02359
   0,33479    0,49003    0,65269    0,97772    0,28322    0,69505    0,70372
   0,30072    0,68705    0,47112    0,68627    0,65030    0,40454    0,63026
   0,15639    0,30391    0,22992    0,69310    0,65716    0,51797    0,68110
   0,72854    0,60188    0,50740    0,74499    0,13459    0,88481    0,12445
   0,80525    0,60180    0,69256    0,74825    0,64388    0,16792    0,45266 

Lets sort the first row of $part, keeping track of original positions:

PS> [ilf64] $indices = 0.0
PS> [ilf64] $sorted = [ilmath]::sort($part['0,1;:'],$indices,0,$false)
PS> $sorted.ToString()
<Double> [2,7]
   0,02006    0,02359    0,08522    0,22956    0,57363    0,59997    0,87217
   0,28322    0,33479    0,49003    0,65269    0,69505    0,70372    0,97772
PS> $indices.ToString()
<Double> [2,7]
         5          6          0          4          3          2          1
         4          0          1          2          5          6          3 

This is all interactive. Of course, we can write complete functions and even complex algorithms that way.
One of the best things: Even in Powershell ILNumerics saves your memory and meets all expectations regarding execution speed. Powershell allows you to consequently use ILNumerics’ typing and scoping rules.

In our feasibility study with Gerd Heber, we show how easy it gets to access an HDF5 file, to convert its data to ILNumerics arrays (implicitly), filter and manipulate a little and even create a full interactive 3D surface graph from it. We demonstrate how to use the type accelerators and to mimic the using statement for artificial scoping. Take a look and let us know, what you think!

The post Putting on a Good Show with HDF5, ILNumerics, and PowerShell appeared first on The ILNumerics Blog.