Reactions

I got a lot of emails after the last two posts (first, second) of people that liked the post and are interested in learning more. Most emails asked me to go into more details about how I did all that. Obviously, not too many people are interested in implementing the particle in a box, so the “what” is not as interesting as the “how”.

In this post I want to focus on the visualization part of the previous two examples and how ILNumerics helped me. As many of our users are scientists and engineers they are typically experts in the computation part but appreciate a little help in the visualization. If you’re interested in the computation part, please be patient until the next post.

The Form

I started by creating a new project in Visual Studio and chose Windows Forms Application. This gives me a vanilla Form1.cs. That’s the basis of all the visualization and helps me – being a scientist and not a coder – a lot, because I couldn’t do all that without the help of Visual Studio in the background.

On the left-hand side in the Toolbox panel I picked TableLayoutPanel under Containers and dragged it into Form1.cs.

I adjusted the table to have three rows and a single column. In the Properties section under Dock I chose Fill so that everything scales when scaling the window later on.

A right click on Table -> Edit Rows and Columns allows you to change the behavior of the three rows. I wanted the first and last row not to scale when changing the window. This is what I chose here.

In the middle row we would like to see two ILNumerics panels. So I dragged another TableLayoutPanel in the middle row and chose two columns and one row. Choosing Dock = Fill for the new table and dragging two ILPanels into the respective columns (again with Dock = Fill) gives us the layout for our final Windows Form.

In the header (the first row) I inserted a FlowLayoutPanel that makes sure that everything in there will be displayed rather neatly. Then I added the labels and the NumericUpDown according to my needs. Please make sure that you go through the properties of every item and choose the values according to your needs. In my case, for instance, the minimum value for the state is 1, since 0 would mean that there is no particle present. Here is the final picture of the form for the 1D particle in a box:

The Wiring

After having the layout ready the interactivity needs to be implemented. In this case there is only one point of user interaction. The user can choose which eigenvector to display. Double-clicking on the NumericUpDown control opens a new tab with a predefined class and a function handling the event when a user changes the value of the eigenvector.

All that’s needed now is to implement the function numericUpDown1_ValueChanged. This is particularly simple in our case. We just call the Update function (still to be implemented) with the new value for the eigenvector.

In the case of the 2D particle in a box we have, in fact, four events that should trigger the Update function. This is easily done by introducing these lines of code in the public function Form1():

     numericUpDown1.ValueChanged += Update;
     numericUpDown2.ValueChanged += Update;
     numericUpDown3.ValueChanged += Update;
     textBox1.TextChanged += Update;

The Implementation

Now, all we need to do is to implement the Update function and the initialization. The initialization makes sure that all the plotting objects are created when starting the program.

     public Form1()
     {
        InitializeComponent();

        var EVID = 1;
        var MeshSize = 1000;

        InitializePanel1(EVID, MeshSize);
        InitializePanel2(EVID, MeshSize);

        label2.Text = string.Format("In appropriate units the energy is {0}", EVID * EVID);
     }

     private void InitializePanel1(int EVID, int MeshSize)
     {
        ILArray<float> XY = ILMath.tosingle(Computing_Module1.CalcWF(EVID, MeshSize));
        var color = Color.Black;
        ilPanel1.Scene =
          new ILScene {
            new ILPlotCube {
              new ILLinePlot (XY, lineColor : color)
            }
          };
     }

The call to Computing_Module1.CalcWF will be explained in the next post. The piece of code above initializes the wave function plot and the density plot in their respective panels with the first eigenvector and a mesh size of 1000.

Finally, the Update function is implemented.

     private void Update(int EVID, int MeshSize = 1000)
     {
       ILArray<float> XY = ILMath.tosingle(Computing_Module1.CalcWF(EVID, MeshSize));
       ilPanel1.Scene.First<ILLinePlot>().Update(XY);
       ilPanel1.Scene.First<ILPlotCube>().Reset();
       ilPanel1.Refresh();

       ILArray<float> XD = ILMath.tosingle(Computing_Module1.CalcDensity(EVID, MeshSize));
       ilPanel2.Scene.First<ILLinePlot>().Update(XD);
       ilPanel2.Scene.First<ILPlotCube>().Reset();
       ilPanel2.Refresh();
       label2.Text = string.Format("In appropriate units the energy is {0}", EVID * EVID);
     }

Again, the CalcWF and CalcDensity calls will be explained in the next post. The Reset() function rescales the axes and the Refresh() function finally puts the new plot on screen.

The post ILNumerics for Science – How did you do the Visualization? appeared first on The ILNumerics Blog.

* Evaluate options to use ILNumerics from IronPython

Several years ago there has been some attempts from ILNumerics users who successfully utilized ILNumerics from within IronPython. But despite our fascination for these attempts, we were not able to catch up and deeply evaluate all options for joining both projects. Years went by and Microsoft has dropped support for IronPython in the meantime. Nevertheless, a considerably large community seems to be active on IronPython. Finally, today is the day I am going to give this a first quick shot.

Disclaimer: I am not a python developer. My experience with IronPython range from 0 … zero (unfortunatley, since the project deserves much more attention). For CPython it is only slighly better. So please bare with me if I went into some stupid direction or if I have completely missed better options in this article! Corrections and suggestions are always warmly welcome.

Setup

I downloaded and installed IronPython from CodePlex. Also, in Visual Studio 2013 there exists a link in the NEW PROJECT dialog advertising the download of IronPython Tools for Visual Studio. I used that to setup Visual Studio for IronPython projects. All setup went smooth and easy. Nice!

Creating Plots with IronPython

The challenge I was interested in the most was how it is possible to utilize the great plotting capabilities of ILNumerics Visualization Engine from IronPython projects. Since matplotlib seems not to be available for IronPython and other alternatives are also pretty rare (if any at all?) having our visualization engine available to IronPython projects seem to be a big improvement.

The good news first: it works and it does so very easily. The following plot is done purely in IronPython:

I started with a fresh new Iron Python Windows Forms Application from the ‘New Project’ dialog in Visual Studio 2013.

This gives a template python file ‘IronPython_ILPanel.py’ with the following content:

import clr
clr.AddReference('System.Drawing')
clr.AddReference('System.Windows.Forms')

from System.Drawing import *
from System.Windows.Forms import *

class MyForm(Form):
    def __init__(self):
        # Create child controls and initialize form
        pass

Application.EnableVisualStyles()
Application.SetCompatibleTextRenderingDefault(False)

form = MyForm()
Application.Run(form)

If you are familiar with the common setup of Windows Forms you may notice some similarities with the stub provided by a new C# Windows Forms Application project. What is otherwise spread over multiple files, now is done all within the global scope of the one and only *.py file: the form class is defined, the static Application class is configured and the event loop is started with an instance of the formerly defined MyForm class. Straight. This corresponds to the common setup for Windows.Forms from any language, where no Designer support exists for. The same could be done in PowerShell, to name just another example.

In order to integrate support for ILNumerics Visualization Engine, I did the following simple steps:

1) If not done so far, go and install ILNumerics Ultimate VS. We offer 30 days trials which contain all features without limitation.
2) Once installed, ILNumerics is available in the “Add Reference” dialog (right click on References in the Solution Explorer). ILNumerics.dll is added from the .NET tab:

3) Make the ILNumerics namespaces available: I added the following lines to the import section at the very top of IronPython_ILPanel.py:

import clr
clr.AddReference('System.Drawing')
clr.AddReference('System.Windows.Forms')
clr.AddReference('ILNumerics')

from System.Drawing import *
from System.Windows.Forms import *
from ILNumerics import *
from ILNumerics.Drawing import *
from ILNumerics.Drawing.Plotting import *

4) Now we are ready to add a plot to the form. In C# we would simply use the ilPanel1_Load event handler which is easily added by help of the forms designer. Since we do not have designer support here, we simply add the setup to the constructor of MyForm:

class MyForm(Form):
   def __init__(self):
     # Create child controls and initialize form
     ilpanel = ILPanel()
     ilpanel.Dock = DockStyle.Fill
     self.Controls.Add(ilpanel)
     # show some demo: first create a plotcube
     pc = ILPlotCube("pc", 0)
     # it will hold a surface with some sinc data in 3D
     # You can use ILRetArray returned from any ILNumerics Computing
     # Module as input argument here directly. Type conversions seem to happen automatically.
     sf = ILSurface(ILSpecialData.sincf(40,50))
     pc.Add(sf)
     # add the plotcube to the scene
     ilpanel.Scene.Add(pc)
     self.Text = "Plotting Fun with ILNumerics in IronPython"

This is all straightforward. The configuration of the panel and the plots is exactly as it would have been done in C# or Visual Basic. However, we have to add the ILPanel to the self.Controls collection manually. The result, of course is exactly the same: a nice interactive form, utilizing OpenGL, with all manipulation options like rotation, pan and zoom. The full spectrum of the Visualization Engine should work out of the box. This includes several, flexible 3D and 2D plotting types as well as the whole scene graph functionality for building your own interactive visualizations.

This is all what needs to be done in order to create nice professional plots directly within IronPython. Press F5 and start your application, rotate the plot with the mouse and learn about all the options you have.

Computing Fun with IronPython and ILNumerics

ILNumerics not only rely on its own flexible n-dimensional array implementation, it also introduces its very own memory management – for very good reasons. In order for the memory managements to work, individual array types (ILArray, ILInArray, ILOutArray and ILRetArray) are used in ILNumerics algorithms. This makes great use of the strict type safety of .NET languages as C# and Visual Basic and of automatic conversions between those types.

Python on the other side is a dynamic language. It does not know the concept of types to the same extend. A straightforward application of the ILNumerics array types is not possible. IronPython, however offers workarounds (clr.Convert) but they are not able to provide the same syntactical convenience as a pure C# algorithm.

My recommendation for the utilization of ILNumerics Computing Engine therefore is as follows:

ILNumerics Computing Engine can be used without restriction. The utilization of existing algorithms is straightforward. Algorithms leveraging the ILNumerics Function Rules can be called directly without any type conversions. The creation of local variables on the other side requires type conversions from ILRetArray to ILArray. This can be done by help of IronPythons clr.Convert function.

The type conversion issue possibly makes it less feasible to write own ILNumerics Computing Engine algorithms in IronPython. But most the time, one would rather want to use existing python algorithms anyway. In order to actually create a new algorithm, one should rather utilize C# and compile the algorithm into its own .NET module which can than easily be imported into your python project and get interfaced from your python code.

Summary

This blog demonstrated how easy it is to utilize ILNumerics from IronPython. Especially the Visualization Engine is incorporated without any problems and offers the full set of visualization and plotting options out of the box. Algorithms created with ILNumerics Computation Engine can directly be interfaced and used – in parallel with numpy algorithms, if the need arise. The syntax of the Computing Engine however is not as expressive and more clumsy due to the absense of implicit type conversions and operator overloads. Complex algorithms one therefore would rather implement in C# or VisualBasic than in IronPython. For small algorithms, like data preprocessing and the like, ILNumerics Computing Engine serves as a faster alternative to numpy arrays.

The post Plotting Fun with ILNumerics and IronPython appeared first on The ILNumerics Blog.

Prerequisites

This comparison is going to be easy and fair! This means, we will not attempt to compare an apple with the same apple, wrapped in a paper bag (like often done with the MKL) nor are we going to use specific features of an individual language/ framework – just to outperform another framework (like using datastructures which are better handled in a OOP language, lets say complicated graph structures or so).

We rather seek for an algorithm of:

• Sufficient size and complexity. A simple binary function like BLAS: DAXPY is not sufficient here, since it would neglect the impact of the memory management – a very important factor in .NET.
• Limited size and complexity in order to be able to implement the algorithm on all frameworks compared (in a reasonable time).

We did chose the kmeans algorithm. It only uses common array syntax, no calls to linear algebra routines (which are usually implemented in Intel’s MKL among all frameworks) and – used on reasonable data sizes – comes with sufficient complexity, computational and memory demands. That way we can measure the true performance of the framework, its array implementation and the feasibility of the mathematical syntax.

Two versions of kmeans were implemented for every framework: A ‘natural’ version which is able to be translated into all languages with minimal differences according execution cost. A second version allows for obvious optimizations to get applied to the code according the language recommendations where applicable. All ‘more clever’ optimizations are left to the corresponding compiler/interpreter.

The test setup: Acer TravelMate 8472TG, Intel Core™ i5-450M processor 2.4GHz, 3MB L3 cache, 4GB DDR3 RAM, Windows 7/64Bit. All tests were targeting the x86 platform.

ILNumerics Code

The printout of the ILNumerics variant for the kmeans algorithm is shown. The code demonstrates only obligatory features for memory management: Function and loop scoping and specific typed function parameters. For clarity, the function parameter checks and loop parameter initialization parts have been abbreviated.

public static ILRetArray<double> kMeansClust (ILInArray<double> X, 
                                             ILInArray<double> k,
                                             int maxIterations, 
                                             bool centerInitRandom,
                                             ILOutArray<double> outCenters) {
   using (ILScope.Enter(X, k)) {

// … (abbreviated: parameter checking, center initializiation)

while (maxIterations --> 0) {
    for (int i = 0; i < n; i++) {
        using (ILScope.Enter()) {
            ILArray<double> minDistIdx = empty(); 
            min(sum(abs(centers - X[full,i])), minDistIdx,1).Dispose();// **
            classes[i] = minDistIdx[0]; 
        }
    }
    for (int i = 0; i < iK; i++) {
        using (EnterScope()) {
            ILArray<double> inClass = X[full,find(classes == i)]; 
            if (inClass.IsEmpty) {
                centers[full,i] = double.NaN;
            } else {
                centers[full,i] = mean(inClass,1);
            }
        }
    }
    if (allall(oldCenters == centers)) break; 
    oldCenters.a = centers.C; 
}
if (!object.Equals(outCenters, null))
    outCenters.a = centers; 
return classes; 
   }

}

The algorithm iteratively assigns data points to cluster centres and recalculates the centres according to its members afterwards. The first step needs n * m * k * 3 ops, hence its effort is O(nmk). The second step only costs O(kn + mn), hence the first loop clearly dominates the algorithm. A version better taking into account available ILNumerics features, would replace the line marked with ** by the following line:

...
min(distL1(centers, X[full, i]), minDistIdx, 1).Dispose();
...

The distL1 function basically removes the need for multiple iterations over the same distance array by condensing the element subtraction, the calculation of the absolute values and its summation into one step for every centre point.

Matlab® Code

For the Matlab implementation the following code was used. Note, the existing kmeans algorithm in the stats toolbox has not been utilized, because it significantly deviates from our simple algorithm variant by more configuration options and inner functioning.

function [centers, classes] = kmeansclust (X, k, maxIterations, 
  centerInitRandom)

% .. (parameter checking and initialization abbreviated)
 
while (maxIterations > 0)
        maxIterations = maxIterations - 1; 
        for i = 1:n
            dist = centers - repmat(X(:,i),1,k); % ***
            [~, minDistIdx] = min(sum(abs(dist)),[], 2);           
            classes(i) = minDistIdx(1); 
        end
        for i = 1:k
            inClass = X(:,classes == i); 
            if (isempty(inClass))
                centers(:,i) = nan;
            else
                centers(:,i) = mean(inClass,2);
                inClassDiff = inClass - repmat(centers(:,i),1,size(inClass,2)); 
            end
        end
        if (all(all(oldCenters == centers))) 
            break;
        end
        oldCenters = centers; 
 end

Again, a version better matching the performance recommendations for the language would prevent the repmat operation and reuse the single column of X for all centres in order to calculate the difference between the centres and the current data point.

...
dist = bsxfun(@minus,centers,X(:,i));
...

FORTRAN Code

In order to match our algorithm most closely, the first FORTRAN implementation simulates the optimized bsxfun variant of Matlab and the common vector expansion in ILNumerics accordingly. The array of distances between the cluster centres and the current data point is pre-calculated for each iteration of i:

subroutine SKMEANS(X,M,N,IT,K,classes) 
!USE KERNEL32
  !DEC$ ATTRIBUTES DLLEXPORT::SKMEANS 

  ! DUMMIES
  INTEGER :: M,N,K,IT
  DOUBLE PRECISION, INTENT(IN) :: X(M,N)
  DOUBLE PRECISION, INTENT(OUT) :: classes(N)
  ! LOCALS 
  DOUBLE PRECISION,ALLOCATABLE :: centers(:,:) &
                   ,oldCenters(:,:) &
                   ,distances(:) & 
                   ,tmpCenter(:) & 
                   ,distArr(:,:)
  DOUBLE PRECISION nan
  INTEGER S, tmpArr(1)
  
  nan = 0
  nan = nan / nan
  
  ALLOCATE(centers(M,K),oldCenters(M,K),distances(K),tmpCenter(M),distArr(M,K))

  centers = X(:,1:K)  ! init centers: first K data points
  do  
    do i = 1, N       ! for every sample...
        do j = 1, K   ! ... find its nearest cluster
            distArr(:,j) = X(:,i) - centers(:,j)         ! **
        end do
        distances(1:K) = sum(abs(distArr(1:M,1:K)),1)    
        tmpArr = minloc ( distances(1:K) )
        classes(i) = tmpArr(1);
    end do
  
    do j = 1,K ! for every cluster 
        tmpCenter = 0; 
        S = 0; 
        do i = 1,N ! compute mean of all samples assigned to it
            if (classes(i) == j) then
                tmpCenter = tmpCenter + X(1:M,i); 
                S = S + 1; 
            end if     
        end do
        if (S > 0) then 
            centers(1:M,j) = tmpCenter / S; 
        else 
            centers(1:M,j) = nan;
        end if 
    end do
    
    if (IT .LE. 0) then ! exit condition
        exit; 
    end if 
    IT = IT - 1; 
    if (sum(sum(centers - oldCenters,2),1) == 0) then
        exit;  
    end if 
    oldCenters = centers; 
  end do
  DEALLOCATE(centers, oldCenters,distances,tmpCenter);
end subroutine SKMEANS

Another version of the first step was implemented which utilizes the memory accesses more efficiently. Its formulation relatively closely matches the ‘optimized’ version of ILNumerics:

    ...
    do i = 1, N          ! for every sample...
        do j = 1, K      ! ... find its nearest cluster
            distances(j) = sum(                                &
                                abs(                           &
                                    X(1:M,i) - centers(1:M,j)))
        end do

        tmpArr = minloc ( distances(1:K) )
        classes(i) = tmpArr(1);
    
    end do
    ...

numpy Code

The general variant of the kmeans algorithm in numpy is as follows:

 
from numpy import *

def kmeans(X,k):
    n = size(X,1)
    maxit = 20
    centers = X[:,0:k].copy()
    classes = zeros((1.,n))
    oldCenters = centers.copy()
    for it in range(maxit):
        for i in range(n):
            dist = sum(abs(centers - X[:,i,newaxis]), axis=0)
            classes[0,i] = dist.argmin()
            
        for i in range(k):
            inClass = X[:,nonzero(classes == i)[1]]
            if inClass.size == 0:
                centers[:,i] = np.nan
            else:
                centers[:,i] = inClass.mean(axis=1)

        if all(oldCenters == centers):
            break
        else:
            oldCenters = centers.copy()

Since this framework showed the slowest execution speed of all implemented frameworks in the comparison (and due to my limited knowledge of numpys optimization recommendations) no improved version was sought.

Parameters

The 7 algorithms described above were all tested against the same data set of corresponding size. Test data were evenly distributed random mumbers, generated on the fly and reused for all implementations. The problem sizes m and n and the number of clusters k are varied according the following table:

	min value 	max value	fixed parameters
m	50		2000		n = 2000, k = 350
n	400		3000		m = 500, k = 350
k	10		1000		m = 500, n = 2000

By varying one value, the other variables were fixed respectively.
The results produced by all implementations were checked for identity. Each test was repeated 10 times (5 times for larger datasets) and the average of execution times were taken as test result. Minimum and maximum execution times were tracked as well.

Results

Ok, I think we make it to the results in this part! The plots first! The runtime measures are shown in the next three figures as error bar plots:

Conclusion

For the optimized versions of kmeans (the stippled lines in the figures) – especially for middle sized and larger data sets (k > 200 clusters or m > 400 dimensions) – the ILNumerics implementation runs at the same speed as the FORTRAN one. This is due to ILNumerics implementing similar optimizations into its builtin functions as the FORTRAN compiler does. Also, the efforts of circumventing around the GC by the ILNumerics memory management and preventing from bound checks in inner loops pay off here. However, there is still potential for future speed-up, since SSE extensions are (yet) not utilized.
For smaller data sets, the overhead of repeated creation of ILNumerics arrays becomes more important, which will be another target for future enhancements. Clearly visible from the plots is the high importance of the choice of algorithm. By reformulating the inner working loop, a significant improvement has been achieved for all frameworks.

The post Fast. Faster …. Performance Comparison: C# (ILNumerics), FORTRAN, MATLAB and numpy – Part II appeared first on The ILNumerics Blog.