Tag Archives: comparison

First Look at Julia on Windows

I recently blogged about the upcoming Lang.NEXT 2012 conference in Redmond. And since the videos are not uploaded yet (and the talk about Julia should pretty much only soon start) I decided to use the time to do some early evaluation of the language with the beautiful suggestive name everyone seems to fall in love immediately. Since we all know how prone love is to projection, I felt I needed a more rational look on the language. And – as usual – as things get clearer you get to know each other more and more and butterflies turn into even more beautiful butterflies … or into something completely different ….

Lets start with some motivation. Julia wants to bridge the gap between established convenient mathematical (prototyping, desktop) systems and high performance computing (parallel) resources. So, basically, it wants to be comfortable and fast. “Huh?” – I hear you say, “this is what ILNumerics does as well!” – and of course you are right. But Julia originates from a very different motivation as ILNumerics. For us, the goal is to provide convenient numeric capabilities with high performance and a comfortable syntax – but to do it directly in a general purpose language. Basically, this brings a lot of advantages when it comes to deployment of your algorithm and it is much easier to utilize all those convenient development tools which are already there for C#. Furthermore, (frequent) transition from business logic to your numerical algorithms can become nasty and error prone.

Julia, on the other side, has to fight other enemies: dynamic language design. Things like dispatching schemes, type inference and – promotion, lexer and parser and certainly a lot more. I really bow to those guys! From a first view they did really succeed. And at the same time, I am glad, that Eric Lippert and his colleagues took away the hard stuff from us. But, of course: by going through all that pain of language design (ok, it sometimes might be fun as well) – you gain the opportunity to optimize your syntax to far less limits. A ‘plus’ of convenience.

Lets take a look at some code. Readers of this blog are already familiar with what turns out to become our favorite algorithm for comparing languages: the kmeans algorithm in its beauty and simplicity. Here comes the Julia version I managed to run on Windows:

function kmeansclust (X, k, maxIterations)

nan_ = 0.0 / 0.0;
n = size(X,2); 
classes = zeros(Int32,1,n); 
centers = rand(size(X,1),k); 
oldCenters = copy(centers); 
while (maxIterations > 0)
        println("iterations left: $maxIterations"); 
        maxIterations = maxIterations - 1;
        for i = 1:n
                Xexp = repmat(X[:,i],1,k);
            	dists = sum(abs(centers - Xexp),1); 
	        classes[i] = find(min(dists) == dists)[1];
        end
        for i = 1:k
            inClass = X[:,find(classes == i)];
            if (isempty(inClass))
                centers[:,i] = nan_;
            else
                centers[:,i] = mean(inClass,2);
            end
        end
        if (all(oldCenters == centers))
            break;
        end
        oldCenters = copy(centers);
 end
 (centers, classes)
end

Did you notice any differences to the Matlab version? They are subtle:

  • Line 29 returns the result as tuple – a return keyword is not required. Moreover, what is returned does not need to be defined in the function definition.
  • Julia implements reference semantics on arrays. This makes the copy() function necessary for assignments on full arrays (-> lines 7 and 27). For function calls this implies, that the function potentially alters its input! Julia states the convention to add a ! to the name of any function, which alters its input parameter.

Besides that, the syntax of Julia can be pretty much compatible to MATLAB® – which is really impressive IMO. Under the hood, Julia even offers much more than MATLAB® scripts are able to do: type inference and multiple dispatch, comprehensions, closures and nifty string features like variable expansion within string constants, as known from php. Julia utilizes the LLVM compiler suite for JIT compilation.

Julia is too young to judge, really. I personally find reference semantics for arrays somehow confusing. But numpy does it as well and nevertheless found a reasonable number of users.

While the above code run after some fine tuning, the current shape of the Windows prebuilt binaries somehow prevented a deeper look in terms of performance. It still needs some quirks and bugs removed. (The Windows version was provided only some hours earlier and it was the first publicly available version for Windows at all.) As soon as a more stable version comes out, I will provide some numbers – possibly with an optimized version (@bsxfun is not implemented yet which renders every comparison unfair). According to their own benchmarks, I would expect Julia to run around the speed of ILNumerics.

Quick and dirty tests and misleading results

Today this blog post popped up on my desktop. It describes a quick attempt to outperform the variance function of our friend library MathNet.Numerics by utilizing the Task Parallel Library from within F#. It obviously worked out – by a factor of 10! Since this seemed strange to me, I couldn’t resist to make my own (very quick and dirty) comparison.

The code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using ILNumerics; 
using MathNet; 
using MathNet.Numerics.Statistics; 
using System.Diagnostics; 

namespace ConsoleApplication6 {
    class Program : ILMath {
        static void Main(string[] args) {
            long ms = 0; 
            int n = 10000000; 
            int it = 100; 

            ILArray<double> A = randn(1,n);
            Stopwatch sw = new Stopwatch(); 
            for (int i = 0; i < it; i++) {
                sw.Restart(); 
                ILArray<double> B = var(A);
                sw.Stop(); 
                ms += sw.ElapsedMilliseconds; 
            }
            Console.WriteLine("ILNumerics needed: {0} ms",ms/(float)it); 
            
            double[] storage = A.ToArray();
            ms = 0; 
            for (int i = 0; i < it; i++) {
                sw.Restart();
                double a = Statistics.Variance(storage); 
                sw.Stop();
                ms += sw.ElapsedMilliseconds;
            }
            Console.WriteLine("Mathnet.Numerics needed: {0} ms", ms / (float)it); 
            Console.ReadKey(); 
        }
    }
}

The result:

ILNumerics needed: 101,87 ms
Mathnet.Numerics needed: 433,67 ms

So the performance of the MathNet implementation is actually not too bad. ILNumerics does parallelize the var() function (the test run on a dual core) and uses unsafe pointer optimized code for the iteration. So a factor of 4 over MathNet is reasonable. I suppose, the speedup of 10 in the referenced F# blog post is more a measure of memory bandwith. The test there have not been repeatedly run and so it appears, the given implementation is advantageous in terms of memory accesses. Possibly some prefetch of cachelines or similar is going on.

Fast. Faster …. Performance Comparison: C# (ILNumerics), FORTRAN, MATLAB and numpy – Part I

“Ehh..! Yet another comparison” you may say. And this is, what it is. Nevertheless, the results might be astonishing…
This post will be split in several parts. Dont know yet, how many. But at least 2. The first part will try to find a little overview or categorization of the conjunction between the terms ‘performance’ and ‘language’.

High Performance – what does it mean?

Several attempts have already been made to measure the impact the .NET CLR introduces to heavy numerical computations. Naturally, this is hard to generalize, since the final execution speed of any program does depend on so many factors. The initial language for the algorithm being only one of them. Among others are important:

  • the set of machine instructions presented to the CPU(s) and how the processor is able to optimize their execution
  • how do the compiler(s) used to get the machine code out of the higher level language are able to support the processor(s) in executing the code fast
  • how do the compiler(s) and/ or the high level algorithm prepare for the utilization of all available computing resources on the executing machine.

“Fast” or “high performance” languages are able to succeed in those points better than those languages, specialized for other tasks than performance. A positive example is FORTRAN, a language designed for other purposes (yet) may be found in JavaScript. At the end, the language (ie. the “features” of the language) are less important than the compilers used to generate executable code. So this is, where we get to the main difference between virtual language environments (Java, .NET and many others) and so called ‘native’ code (C/C++, FORTRAN being the most popular). While the native fraction does typically utilize one compiler to generate the final machine code in one step, the virtual fraction does introduce at least two steps. A language specific compiler generating “byte code” (‘Intermediate Language’) at development time and a Just in Time (JIT) compiler transforming that byte code into machine instructions right before the execution.

‘Virtually High’ Performance

It is a common (marketing?) argument, that the JIT compiler of a virtual machine is able to optimize the final code much better, since at runtime the actual system resources (model and number of CPU(s), cache sizes etc.) are well known. In reality, the speed of programs written for virtual machines suffer from the lack of implemented optimizations. The reason might be found in the intended audience for those environments: enterprise applications simply do not need numberchrunching performance!

Nevertheless, many attempts exist from the scientific community to utilize virtual runtime environments for numerical computing. Due to its maturity most of them aim towards Java [1],[2],[3]. Attempts for .NET just like most attempts for Java aim at providing wrapper classes in order to make the performance of ‘native’ libraries available to the higher level language. And there is nothing dubious about it. Developers writing algorithms in ‘native’ languages (lets say FORTRAN) are well advised to utilize those libraries as well. Everyone would have a hard time to beat their performance by writing – lets say – a faster matrix multiplication than the one provided by the Intel MKL (just to name one). I recently looked into exact that topic and will possibly publish those results in a later post.

So in order to categorize languages after their suitability for scientific computing, we identify the following needs:

  • The language should provide a ‘nice’ syntax to the user. This includes things like multidimensional arrays and overloaded operators.
  • There must exist tools (like compilers and/or other supporting infrastructure, possibly a library) which support the fast execution on a wide range of machines.

For a very long time, FORTRAN has been the best language according to those goals. But recent changes in the computer architecture show a significant shift in the responsibility to care about performance. While earlier, software developers commonly relied on the processor (or their designers) to run any code as fast as possible, nowadays this is over. There will be no significant improvement in a single processing unit in upcoming design cycles. Major improvements will be made by utilizing thread level performance, ie. the utilization of multicore machines and GPU power. But it will be the responsibility of the software layer now, to utilize those resources.
It wouldn’t be the ILNumerics Developer Blog, if we wouldn’t believe, that .NET and C# have very good potential to fullfill those needs – possibly better than any other major language around. In terms of the syntactic sugar the proove has been done already – simply download and try it yourself if you haven’t done yet! But ILNumerics also fullfills the second requirement! The .NET CLR as well as the mono runtime do offer several options to utilize computing resources better than other virtual machines.

Ok… but where are the plots?

In the next part I will give the results of speed measure of an every day algorithm: the k-means clustering algorithm. I have seen ‘comparisons’ on the web, where a library, utilizing the MKL compares its own matrix multiplication (carried out in the MKL) with the one from the MKL!?! Dont know, who is going to be fooled that way. We will not compare anything not executed within the virtual machine. No matrix multiplication, no linear algebra. Those parts do use optimized tools and hence run at highest possible speed anyway. K-means on the other side does only utilize array features: distance computations, binary operators, aggregate function (sum, min) etc. – all carried out within the corresponding framework.

We did choose the most popular representatives of scientific computing languages: Matlab, FORTRAN, numpy and ILNumerics ;) and wrote the kmeans algorithm for each language twice: one time while having the best fairness in between the languages in mind and one time with major optimizations the specific language provides. Results coming in the next part…

Comparison .NET JAVA C++ and FORTRAN

I recently stumbled on a plot provided by indeed.com:

I found it interesting that .NET even seems to be leading in the past and only recently Java had catched up with .NET. Somehow, I always thought it was the other way around. Also, Fortran does not seem to be visible at all. Most likely, because you wouldn’t enter any position because “you know Fortran”. Rather Fortran is somehow a requirement which you would need to learn in case your position requires it, I suppose.