How to test static method call - Benchmarks

This is a continuation of my last post.

For a long time I rejected to even test Delegates as a method to make static method calls "virtual" because of performance. My mind doesn't allow me to accept the fact that they can be viable alternative.

But real numbers was a little strange.

First, test console program

class Program : IProgram
    {
        private static Action TestStaticDelegate = TestStatic;
        private static Action TestStaticNoInlineDelegate = TestStaticNoInline;

        static void Main(string[] args)
        {
            TestStatic();
            TestStaticNoInline();
            IProgram p = new Program();
            Action TestInterfaceDelegate = p.TestInterface;
            p.TestInterface();

            int number = 1000000000;
            
            DateTime start = DateTime.Now;
            for (int i = 0; i < number; i++ )
                TestStatic();
            Console.WriteLine("TestStatic: " + (DateTime.Now - start).Ticks);

            start = DateTime.Now;
            for (int i = 0; i < number; i++)
                TestStaticNoInline();
            Console.WriteLine("TestStaticNoInline: " + (DateTime.Now - start).Ticks);

            start = DateTime.Now;
            for (int i = 0; i < number; i++)
                p.TestInterface();
            Console.WriteLine("TestInterface: " + (DateTime.Now - start).Ticks);

            start = DateTime.Now;
            for (int i = 0; i < number; i++)
                TestStaticDelegate();
            Console.WriteLine("TestStaticDelegate: " + (DateTime.Now - start).Ticks);

            start = DateTime.Now;
            for (int i = 0; i < number; i++)
                TestStaticNoInlineDelegate();
            Console.WriteLine("TestStaticNoInlineDelegate: " + (DateTime.Now - start).Ticks);

            start = DateTime.Now;
            for (int i = 0; i < number; i++)
                TestInterfaceDelegate();
            Console.WriteLine("TestInterfaceDelegate: " + (DateTime.Now - start).Ticks);
        }

        private static int x = 0;

        static void TestStatic()
        {
            x++;
        }

        private static int y = 0;
        [MethodImpl(MethodImplOptions.NoInlining)]
        static void TestStaticNoInline()
        {
            y++;
        }

        private int z = 0;
        public void TestInterface()
        {
            z++;
        }        
    }

    public interface IProgram
    {
        void TestInterface();
    }

As you can see I want to compare static method calls with delegates and (the most important) with virtual call.
I've run this program on my AMD Athlon XP 2400+ 2GHz machine (it was Release version of course :-). The results were following:

TestStatic: 18326352
TestStaticNoInline: 39556880
TestInterface: 56280928
TestStaticDelegate: 67697344
TestStaticNoInlineDelegate: 61989136
TestInterfaceDelegate: 56180784

For me the most intresting numbers are TestInterface and TestStaticDelegate, because it compares Dependency Injection using interface with DI using Delegates. As you can see delegates are around 17% slower then interface method call. That was predictable but I've suspected much higher difference. I've run this test few times and the results was always similiar.

According to the power this method gives me, it's acceptable performance penalty, but for some strange reason I've thought that it will be interesting to test it on Intel platform also, so I've run it on my work machine with Intel Pentium 4 3 Ghz. The results were little strange:

TestStatic: 17031359
TestStaticNoInline: 61250392
TestInterface: 55000352
TestStaticDelegate: 51250328
TestStaticNoInlineDelegate: 49844069
TestInterfaceDelegate: 34218969

Woow, it seems that on Intel, Delegates are something around 7% faster then interface call. What's more, delegate instance call is 40% faster then interface one.

I've tried to find some bug in this code, but everything looks, fine for me. Event tried to change (DateTime.Now - start).Ticks to (DateTime.Now - start).ToString() but results were similiar:

TestStatic: 00:00:01.6875108
TestStaticNoInline: 00:00:06.0625388
TestInterface: 00:00:05.5312854
TestStaticDelegate: 00:00:05.0625324
TestStaticNoInlineDelegate: 00:00:05.2187834
TestInterfaceDelegate: 00:00:03.4687722

Seeing this, I saw also that even on AMD, delegate instance call (TestInterfaceDelegate) is faster then normal interface call (TestInterface).

 

My Conclusions

• If you need performance use static methods and TypeMock for testing



• If you need to inject some code at runtime you can freely use Delegates without any significant performance loose. Even better - it looks like .NET 3.5 JIT promotes functional programing (static methods and delegates) by making it more performant than object oriented (interface calls) one - at least on Intel platform.



• If you don't wan't to use TypeMock you can safely use Delegates based Dependency Injection.

How to test static method call in c# without TypeMock

If you think - like me - that TDD is not only testing, but also design process, you probably are using Rhino Mocks or any other mocking tool that is based on interface/virtual methods implementation. But how can one mock a call to static method like

File.Open("...", FileMode.Open);

The only method until now (for me) was to use Typemock. But until you buy a paid version, you cannot use it without typing method names as string - which I don't like.

After long long long searching I think I found an alternative - Delegates.

Lets see an example (in c# 3.5):

    public class SomeClass
    {
        public Func<string, FileMode, Stream> FileOpen = File.Open;

        public void DoSomethingWithFile(string path)
        {
            using (Stream s = FileOpen(path, FileMode.Open))
            {
                //...
                //...
            }
        }
    }

As you see, I placed a public delegate to File.Open which I can change later in my test

        [Test]
        public void TestDoSomethingWithFile()
        {
            SomeClass sc = new SomeClass();
            string testFile = Path.GetTempFileName();
            //...fill this testFile

            sc.FileOpen = delegate { return File.OpenRead(testFile); };
            sc.DoSomethingWithFile(testFile);
            
            //Asserts here
        }

What is great about such pattern is that you can treat those delegates in class as references in dll file.
Another great thing is that you can use Dependency Injection to inject static(or not) methods to the class.

 

And thats it. So simple right?
Ummm, but what about performance, delegates can't work as fast as static calls.
Well, I'll show you some benchmarks later.

Welcome

So the time has come for me also :-)

It's time to try to archive some of my thougths about programming and whatever comes to my mind.

So let's start and see what will come from this :-)

 

First of all (and this is the only time I will do it) I want to apologize for my English. I'm not a native English speaker so be ready for mistakes :-)