HDF5 is a file format (Hierarchical Data Format) especially desgined to handle huge amount of numerical data. Just to mention an example,  NASA chose it to be the standard file format for storing data from the Earth Observing System (EOS).

ILNumerics easily handles HDF5 files. They can be used to exchange data with other software tools, for example Matlab mat files. In this post I will show a step by step guide – how to interface ILNumerics with Matlab.

Save Matlab data to HDF5 format

Let’s start by creating some arbitrary data with Matlab and saving it in HDF5 format, Matlab supports it from 2006 September, version 7.3.

matrix = randn(10, 10) * 5;
vector = [1 2 3 4 5]';
carr = 'It is a char array!';
noise = rand(1,100) - rand(1, 100);
sinus = 10 * sin(linspace(0, 2*pi, 100)) + noise;
complex = rand(1, 5) * (10 + 10i);
save('test_hdf5.mat', '-v7.3');

Load Matlab data with ILNumerics

Let’s now see what ILNumerics sees

Matlab simply stored all arrays as individual datasets in the root node of the HDF5 file. It is now easily possible to access the data! We could take a peek and inspect them right within the Visual Studio debugger visualizers. But let’s see how the data is loaded programmatically:

// Reading from Matlab
using (H5File file = new H5File("test_hdf5.mat"))
{
    using (ILScope.Enter())
    {
        ILArray<double> matrix = file.Get<H5Dataset>("matrix").Get<double>();
        ILArray<double> vector = file.Get<H5Dataset>("vector").Get<double>();
        ILArray<char> carr = file.Get<H5Dataset>("carr").Get<char>();
        ILArray<double> noise = file.Get<H5Dataset>("noise").Get<double>();
        ILArray<double> sinus = file.Get<H5Dataset>("sinus").Get<double>();
    }
}

The datasets are loaded – and the whole process was very easy. It took me less than ten minutes to figure out the methods. It would have been even less if I had take a look on the examples of ILNumerics

Now, let’s startup the Array Visualizer! Just write ‘array’ in the Quick Launch box of Visual Studio, or open a new visualizer window via VIEW -> OTHER WINDOWS -> Array Visualizer. In the visualizer expression field enter: ‘sinus’. The array is now visualized, how cool is that

Visualizing Matlab mat files – on the fly

Now we have shown the content of the variable ‘sinus’. Just as well, we could visualize the content of the HDF5 file and have the visualizer follow any changes to it immediately:

If we multiply it with -1, the expression is immediately evaluated:

And what is even more interesting, if we change the HDF5 file interactively, in the immediate window, the ILNumerics Array Visualizer reflects that on the fly:

Loading ILArrays into Matlab

Now let us check, what happens if we want to go from ILNumerics to Matlab. Saving a noise cosine with the following code:

// Writing some data to HDF5
// Note: It will not make it a proper mat file, only HDF5!
if (File.Exists("test_iln.mat"))
    File.Delete("test_iln.mat");

using (H5File newFile = new H5File("test_iln.mat"))
{
    using (ILScope.Enter())
    {
        ILArray<double> noisyCos = ILMath.cos(ILMath.linspace(0, 3 * Math.PI, 1000)) * 10 + ILMath.rand(1, 1000);
        H5Dataset set = new H5Dataset("noisyCos", noisyCos);
        newFile.Add(set);
    }
}

Since the ILNumerics HDF5 is not a mat file format, the only way to load it (as for now) is with one of the ‘h5*’ methods of Matlab. To check the info and load, the following code is used:

h5info('test_iln.mat', '/')
noisyCos = h5read('test_iln.mat', '/noisyCos');

In this way, it is possible to load ILArrays into Matlab

Altering Matlab mat files in ILNumerics

It is possible to alter loaded mat files with some limitations. What I found is that the length of the array can not be changed. Obviously, Matlab mat files are created as datasets without chunks, so the dataspace size cannot be changed after the file was created. However, one can alter the data (without changing the size) :

// Altering data of mat file (v7.3 format)
using (H5File file = new H5File("test_hdf5.mat"))
{
    using (ILScope.Enter())
    {
        ILArray<double> sinus = file.Get<H5Dataset>("sinus").Get<double>();
        sinus.a = sinus * -1;
        file.Get<H5Dataset>("sinus").Set(sinus, "0;:");
    }
}

Let me draw your attention to the use of the Set function here. I first tried to use:

file.Get<H5Dataset>("dsname").Set(A);

But this would attempt to overwrite the whole dataset – something which could potentially change the size, hence would require a chunked dataset. Therefore, I figured that I have to be more specific and need to provide the target data range also:

file.Get<H5Dataset>("sinus").Set(sinus, "0;:")

More Limitations

Out of curiosity I checked the complex array, to see if it is okay.

Unfortunately it throws a H5Exception, this is something I will look into and work on. Fixing this problem will require support for compound datatypes in our HDF5 API.

Try out the example

To play with it, head to the examples and download the code:
Reading and Writing Matlab data with ILNumerics

Conclusion

To sum up, it is possible and easy to load data stored in Matlab mat files to ILNumerics with the HDF5 IO handling, and it is also possible to load ILArrays to Matlab as well.

I am opened to suggestions and feedback, looking forward to hearing from you! Share your experiences with us, so we can learn from you!

The post HDF5 and Matlab Files – Fun with ILNumerics appeared first on The ILNumerics Blog.

Find all quick reference charts in our Quick Start Section.

The post ILNumerics for Matlab® Users (Update) appeared first on The ILNumerics Blog.

High Performance – what does it mean?

Several attempts have already been made to measure the impact the .NET CLR introduces to heavy numerical computations. Naturally, this is hard to generalize, since the final execution speed of any program does depend on so many factors. The initial language for the algorithm being only one of them. Among others are important:

• the set of machine instructions presented to the CPU(s) and how the processor is able to optimize their execution
• how do the compiler(s) used to get the machine code out of the higher level language are able to support the processor(s) in executing the code fast
• how do the compiler(s) and/ or the high level algorithm prepare for the utilization of all available computing resources on the executing machine.

“Fast” or “high performance” languages are able to succeed in those points better than those languages, specialized for other tasks than performance. A positive example is FORTRAN, a language designed for other purposes (yet) may be found in JavaScript. At the end, the language (ie. the “features” of the language) are less important than the compilers used to generate executable code. So this is, where we get to the main difference between virtual language environments (Java, .NET and many others) and so called ‘native’ code (C/C++, FORTRAN being the most popular). While the native fraction does typically utilize one compiler to generate the final machine code in one step, the virtual fraction does introduce at least two steps. A language specific compiler generating “byte code” (‘Intermediate Language’) at development time and a Just in Time (JIT) compiler transforming that byte code into machine instructions right before the execution.

‘Virtually High’ Performance

It is a common (marketing?) argument, that the JIT compiler of a virtual machine is able to optimize the final code much better, since at runtime the actual system resources (model and number of CPU(s), cache sizes etc.) are well known. In reality, the speed of programs written for virtual machines suffer from the lack of implemented optimizations. The reason might be found in the intended audience for those environments: enterprise applications simply do not need numberchrunching performance!

Nevertheless, many attempts exist from the scientific community to utilize virtual runtime environments for numerical computing. Due to its maturity most of them aim towards Java [1],[2],[3]. Attempts for .NET just like most attempts for Java aim at providing wrapper classes in order to make the performance of ‘native’ libraries available to the higher level language. And there is nothing dubious about it. Developers writing algorithms in ‘native’ languages (lets say FORTRAN) are well advised to utilize those libraries as well. Everyone would have a hard time to beat their performance by writing – lets say – a faster matrix multiplication than the one provided by the Intel MKL (just to name one). I recently looked into exact that topic and will possibly publish those results in a later post.

So in order to categorize languages after their suitability for scientific computing, we identify the following needs:

• The language should provide a ‘nice’ syntax to the user. This includes things like multidimensional arrays and overloaded operators.
• There must exist tools (like compilers and/or other supporting infrastructure, possibly a library) which support the fast execution on a wide range of machines.

For a very long time, FORTRAN has been the best language according to those goals. But recent changes in the computer architecture show a significant shift in the responsibility to care about performance. While earlier, software developers commonly relied on the processor (or their designers) to run any code as fast as possible, nowadays this is over. There will be no significant improvement in a single processing unit in upcoming design cycles. Major improvements will be made by utilizing thread level performance, ie. the utilization of multicore machines and GPU power. But it will be the responsibility of the software layer now, to utilize those resources.
It wouldn’t be the ILNumerics Developer Blog, if we wouldn’t believe, that .NET and C# have very good potential to fullfill those needs – possibly better than any other major language around. In terms of the syntactic sugar the proove has been done already – simply download and try it yourself if you haven’t done yet! But ILNumerics also fullfills the second requirement! The .NET CLR as well as the mono runtime do offer several options to utilize computing resources better than other virtual machines.

Ok… but where are the plots?

In the next part I will give the results of speed measure of an every day algorithm: the k-means clustering algorithm. I have seen ‘comparisons’ on the web, where a library, utilizing the MKL compares its own matrix multiplication (carried out in the MKL) with the one from the MKL!?! Dont know, who is going to be fooled that way. We will not compare anything not executed within the virtual machine. No matrix multiplication, no linear algebra. Those parts do use optimized tools and hence run at highest possible speed anyway. K-means on the other side does only utilize array features: distance computations, binary operators, aggregate function (sum, min) etc. – all carried out within the corresponding framework.

We did choose the most popular representatives of scientific computing languages: Matlab, FORTRAN, numpy and ILNumerics and wrote the kmeans algorithm for each language twice: one time while having the best fairness in between the languages in mind and one time with major optimizations the specific language provides. Results coming in the next part…

The post Fast. Faster …. Performance Comparison: C# (ILNumerics), FORTRAN, MATLAB and numpy – Part I appeared first on The ILNumerics Blog.