Memory management for ILNumerics.Net

In this section we take a look into how ILNumerics.Net optimizes memory usage internally.

.NET applications are using a managed heap for managing memory. They are using a garbage collector which is responsible for freeing objects not referenced from any "root" object anymore. It is well known, that this whole process comes with advantages and disadvantages compared to the "traditional way" used in unmanaged applications (f.e. C++).

• Developers dont have to carefully free every object ever created. Memory leaks are not possible anymore - at least from this naiv view.
• Memory allocations (in C# with the new keyword) usually will be much faster than in unmanged environments. This is related to the way the heap is managed without 'wholes' in it.
• Engineers of the .NET framework are expected to keep on developing more and more sophisticated algorithms for the heap management in the future.

• The garbage collector introduces a measurable performance hit on the application. Once all memory available has been reserved by the application, the garbage collector starts freeing memory for unused objects. Therefore the whole object hirarchy has to be walked once. Even if there are some tweaks implemented (f.e. generations), the performance overhead for this process cannot be neglected.
• The moment when objects are actually unregistered from memory is not in any way deterministic. Since memory for objects can only get reclaimed after the next garbage collector run, determining the amount of memory an application uses at a specific point of execution is somehow difficult. When the GC does starts its tasks is usually managed by the framework. However, developers are not helpless, as we will see later on.

How ILNumerics.Net optimizes memory usage

The keys for optimizing memory are: avoidance and recycling. Avoidance is done in ILNumerics.Net to the most extend possible by carefully designing its classes and methods to use the most memory effective version possible. A good example here is the capablity to create true referencing arrays. Since this is all about your data and those data will need to be stored and computed with, we also must exploit the second way: recycling.

ILNumerics.Net internally uses a memory pool serving as temporary storage for large objects like System.Arrays. Those arrays are the underlying storage type for all ILArrays. Whenever possible storage to be cleaned up is placed into the pool for later reusing. Attempts to create new arrays is than delegated to a pool method. If a suitable array was found in the pool it is used, otherwise it is newly created.

The following figure demonstrates the livecycle of System.Array objects stored and reclaimed from the memory pool.

Livecylce for System.Arrays. After creation the arrays resides in the pool for a while, ready for beeing reused.

Whenever possible objects will be reclaimed from the pool. This will clearly decrease the performance pressure to the garbage collector, since the object does not have to be created from the heap, but simply 'returned from parking' in the pool. The number of garbage collector runs decreases. Of course only objects can be reclaimed, which have been placed into the pool before. But how can this be done, without explicitly knowing, when the object is not used anymore? The answer is again: the garbage collector.

How objects are registered into the pool

In many expressions arrays are implicitly created without giving the developer direct access to its handle. Take the following expression as example:

ILArray<double> A = ILMath.zeros(200,100,10) + 2.0; 

Here actually two ILArray<double>s will get created: One is returned from ILMath.zeros and the other is returned from ILMath.Add(). The first one is given to the ILMath.Add function and after returning from it, its handle is lost. Fortunately the garbage collector still keeps a reference to it. When the next run of the GC is started, a finalization method is called for the object which will save its inner System.Array into the ILNumerics.Net memory pool. However the object will therefore only be available in the pool after the next GC run! Consider the following example:

ILArray<double> A = (ILMath.zeros(200,100,10) + 2.0) / 4.0; 

This time three arrays are created, each returning from ILMath.zeros(), ILMath.add() and ILMath.divide() respectively. The first two arrays handles are not accesible to the developer. Only the handle for the arrays returnd by ILMath.divide() will be stored into A. Let's take a closer look to what happens exactly.

After the first array returned from ILMath.zeros() has been given to the ILMath.add() funtion, it is neither used nor referenced anymore. Wouldn't it be great, if we could reuse its memory for the next function ILMath.divide()? Unfortunately, like stated above, there is no good way of telling, when the garbage collector comes around to save the array into the pool. Most likely the 3rd function will be called before the first array has been set free and therefore cannot get reclaimed in time.

In the article optimizing memory usage we will discover strategies to get around this situation.

How objects are reclaimed from the pool

ILNumerics.Net by default will always try to reuse objects registered to the pool. Any request for memory for arrays to be created is therefore delegated to the memory pool. This is true for all situations:

• creation of array from the scratch (zeros,ones,rand, ILArray<> constructors etc.)
• subarray creation, if the result is a new solid array (mostly vectors)
• all mathematical functions and operators
• transposes and clone operations

Reusing objects in the manner described above can drastically diminish the amount of memory used. Read about detailed informations how to tune the memory pool for your specific application in the article about optimizing memory.