I recently ran across a nasty scenario that illustrated to me how dangerous following any fixed rules for optimization can be…

The rules in C# when optimizing code are very different than C or C++.  Often, they’re exactly backwards.  For example, in C and C++, lifting a variable out of loops in order to avoid memory allocations often can have huge advantages.  If some function within a call graph is allocating memory dynamically, and that gets called in a loop, it can dramatically slow down a routine.

This can be a tricky bottleneck to track down, even with a profiler.  Looking at the memory allocation graph is usually the key for spotting this routine, as it’s often “hidden” deep in call graph.  For example, while optimizing some of my scientific routines, I ran into a situation where I had a loop similar to:

for (i=0; i<numberToProcess; ++i)
{
   // Do some work
   ProcessElement(element[i]);
}

This loop was at a fairly high level in the call graph, and often could take many hours to complete, depending on the input data.  As such, any performance optimization we could achieve would be greatly appreciated by our users.

After a fair bit of profiling, I noticed that a couple of function calls down the call graph (inside of ProcessElement), there was some code that effectively was doing:

// Allocate some data required
DataStructure* data = new DataStructure(num);
// Call into a subroutine that passed around and manipulated this data highly
CallSubroutine(data);
// Read and use some values from here
double values = data->Foo;
// Cleanup 
delete data;
// ...
return bar;

Normally, if “DataStructure” was a simple data type, I could just allocate it on the stack.  However, it’s constructor, internally, allocated it’s own memory using new, so this wouldn’t eliminate the problem.  In this case, however, I could change the call signatures to allow the pointer to the data structure to be passed into ProcessElement and through the call graph, allowing the inner routine to reuse the same “data” memory instead of allocating.  At the highest level, my code effectively changed to something like:

DataStructure* data = new DataStructure(numberToProcess);
for (i=0; i<numberToProcess; ++i)
{
   // Do some work
   ProcessElement(element[i], data);
}
delete data;

Granted, this dramatically reduced the maintainability of the code, so it wasn’t something I wanted to do unless there was a significant benefit.  In this case, after profiling the new version, I found that it increased the overall performance dramatically – my main test case went from 35 minutes runtime down to 21 minutes.  This was such a significant improvement, I felt it was worth the reduction in maintainability.

In C and C++, it’s generally a good idea (for performance) to:

• Reduce the number of memory allocations as much as possible,
• Use fewer, larger memory allocations instead of many smaller ones, and
• Allocate as high up the call stack as possible, and reuse memory

I’ve seen many people try to make similar optimizations in C# code.  For good or bad, this is typically not a good idea.  The garbage collector in .NET completely changes the rules here.

In C#, reallocating memory in a loop is not always a bad idea.  In this scenario, for example, I may have been much better off leaving the original code alone.  The reason for this is the garbage collector.  The GC in .NET is incredibly effective, and leaving the allocation deep inside the call stack has some huge advantages.  First and foremost, it tends to make the code more maintainable – passing around object references tends to couple the methods together more than necessary, and overall increase the complexity of the code.  This is something that should be avoided unless there is a significant reason.  Second, (unlike C and C++) memory allocation of a single object in C# is normally cheap and fast.  Finally, and most critically, there is a large advantage to having short lived objects.  If you lift a variable out of the loop and reuse the memory, its much more likely that object will get promoted to Gen1 (or worse, Gen2).  This can cause expensive compaction operations to be required, and also lead to (at least temporary) memory fragmentation as well as more costly collections later.

As such, I’ve found that it’s often (though not always) faster to leave memory allocations where you’d naturally place them – deep inside of the call graph, inside of the loops.  This causes the objects to stay very short lived, which in turn increases the efficiency of the garbage collector, and can dramatically improve the overall performance of the routine as a whole.

In C#, I tend to:

• Keep variable declarations in the tightest scope possible
• Declare and allocate objects at usage

While this tends to cause some of the same goals (reducing unnecessary allocations, etc), the goal here is a bit different – it’s about keeping the objects rooted for as little time as possible in order to (attempt) to keep them completely in Gen0, or worst case, Gen1.  It also has the huge advantage of keeping the code very maintainable – objects are used and “released” as soon as possible, which keeps the code very clean.  It does, however, often have the side effect of causing more allocations to occur, but keeping the objects rooted for a much shorter time.

Now – nowhere here am I suggesting that these rules are hard, fast rules that are always true.  That being said, my time spent optimizing over the years encourages me to naturally write code that follows the above guidelines, then profile and adjust as necessary.  In my current project, however, I ran across one of those nasty little pitfalls that’s something to keep in mind – interop changes the rules.

In this case, I was dealing with an API that, internally, used some COM objects.  In this case, these COM objects were leading to native allocations (most likely C++) occurring in a loop deep in my call graph.  Even though I was writing nice, clean managed code, the normal managed code rules for performance no longer apply. 

After profiling to find the bottleneck in my code, I realized that my inner loop, a innocuous looking block of C# code, was effectively causing a set of native memory allocations in every iteration.  This required going back to a “native programming” mindset for optimization.  Lifting these variables and reusing them took a 1:10 routine down to 0:20 – again, a very worthwhile improvement.

Overall, the lessons here are:

• Always profile if you suspect a performance problem – don’t assume any rule is correct, or any code is efficient just because it looks like it should be
• Remember to check memory allocations when profiling, not just CPU cycles
• Interop scenarios often cause managed code to act very differently than “normal” managed code.
• Native code can be hidden very cleverly inside of managed wrappers

Understanding Big-O notation is critical in making good decisions about algorithms or libraries used to implement routines.  However, there is often a lot of misunderstanding about what it means, and I frequently see this used as justifications for poor decisions.

First, let’s define Big-O notation.  Big-O notation describes the limiting factor of a function as the argument or domain size of the function approaches a specific value, typically infinity.  Big-O notation describes a growth rate, not a speed.  This is important – one very common misunderstanding about Big-O notation is that it describes how “fast” a function will be – and this is never true.  It describes how a function or algorithm will scale, but says absolutely nothing about speed.

When programming, Big-O notation can be used to describe many different things.  Most frequently it is describing the algorithm in terms of instruction count or speed, but can also be used to describe memory or resource usage.  It’s important to know what is being described when making decisions based off this information, and seeing that an algorithm is “O(N)” isn’t enough information, in and of itself, to know what that means.

When and why is this useful?  It becomes very important to understand this when we start dealing with collections, especially if the collection is large or of an unknown scale.  For example, say you’re going to do some basic image processing, and you have the choice between a couple of different algorithms.  If each algorithm’s limiting behavior is based off the total image size, and the algorithm’s operation count is described in Big-O notation, the total number of operations will vary dramatically as your image size changes.  For example, let’s say we have the choice of 3 algorithms described by three different, but common Big-O notations:

• Algorithm A: Linear – O(N)
• Algorithm B: Logarithmic – O(log N)   
• Algorithm C: Quadratic – O(N^2)

[I am, for here, assuming Log in base 10, but for Big-O, it could be any logarithmic base and still be O(log N)]

Note the huge discrepancy in terms of instruction count!  Algorithm C is a quadratic algorithm, meaning that its instruction count increases by the square of the number of elements (pixels, in this case).

Now, given the information above, it seems like you would naturally always choose Algorithm B – it scales much, much better.  Often, this is true, and this is a good way to choose between algorithms, but we don’t have enough information to make an educated decision at this point.

Remember above – I mentioned that Big-O describes a growth rate, and not a speed.  In addition, it describes an upper bound for the growth rate, not necessarily the average case scenario.  For example, it may be that algorithm C is the fastest algorithm for processing a single pixel – perhaps by a factor of thousands.  Let’s look at these algorithms, and assign the number of actual microseconds each pixel takes to process on average, and look at the same chart as above:

• Algorithm A: 0.1 seconds
• Algorithm B: 10 seconds
• Algorithm C: 50 microseconds

This shows off a basic misconception of Big-O notation.  It’s not about speed.  Granted, if you’re planning to deal with megapixel imagery, you’d probably still choose the algorithm above with the best Big-O notation.  However, there are two things that may trip you up here when you’re developing this algorithm.

1. If you developed the three algorithms above looking only at a small image, you’d probably have picked algorithm C, hands down, because the per-pixel routine is so dramatically faster.
2. If you developed the three algorithms above with a large image, you’d have given up on algorithm C quickly, because of it’s poor scalability.

However, both of those are not necessarily good things.

In case 1: The reason this is bad is obvious.  Later, when you fed the routine a large image, you’d be in trouble (especially if you didn’t realize that it was quadratic at the time).

In case 2: If this routine was always going to be used on very small images, Algorithm C is a superior choice, even though it has the worst Big-O description.  This is because it’s single element speed is so much better. 

Also, Big-O only describes the upper bound (which is very important), but the average case of an O(N^2) function may actually be closer to N Log N (quicksort is this way).  It may be that, when we actually look at real execution times in a profiler with real data, the algorithm performs, on average, in N log (N), in which case the “real” runtime may be more like:

All of a sudden, our O(N^2) algorithm is looking very, very good – much better than any of our other algorithms!

What can we take from this?  There are a couple of important points:

• Big-O Notation can provide critical information about the scalability of an algorithm.  Coupled with information about the algorithm itself, this can help us decide which of multiple algorithms will be the “best” choice.
• Big-O Notation does not provide information about speed.  Factors in the algorithm itself, including total number of instructions required, are not included as part of this notation.
• Knowing the Big-O notation for a method, or doing your own analysis of the algorithm, is not a substitute for profiling.  “Bad” routines in terms of their Big-O notation can outperform “good” routines, in certain situations.