Programming Language Benchmarks

The relative performance of different programming languages has always been of great interest to me. I began my professional career back in 1986 when I worked as a developer on a now defunct commercial simulation product called See Why. See Why (developed at the time by a division of Istel Ltd. that is now better known today as Lanner Group) was a library of FORTRAN 77 routines that supported the development of visual, interactive discrete-event simulation models.

See Why was the original foundation of the still-popular Witness simulation application. At that time, Witness was a front-end to See Why that was also programmed in FORTRAN 77.

FORTRAN 77 was widely regarded at the time as the highest performing high-level language available - largely credited to the optimizations that its compilers were capable of. (It should be remembered that the performance snobs of the day waxed lyrical about assembly language and looked down on all high-level languages as being inherently inefficient. Fortunately, programming languages, hardware and attitudes have evolved somewhat since. So much so that C is now often regarded as a low-level programming language, when I still regard it very much as a high-level language.)

FORTRAN 77 had no support for manipulating memory dynamically, which meant that See Why had to claim a fixed, static amount of memory (stored in an array called the Simulation Array) at start-up and use that to store all required simulation elements (entities, vessels, sets, timeseries, histograms and user-defined distributions). This restricted how many simulation elements you could have. If you needed more elements than would fit into the Simulation Array (regardless of how much memory, virtual or otherwise, your machine had), then that was just tough. (The size of the Simulation Array was a meagre 32 KB in See Why A or 64 KB in See Why B.)

FORTRAN 77 not only had no support for object-oriented programming, it didn't even support data structures! With hardware (a top-of-the-range PC at the time ran with an 8 MHz i286 processor and had 640 KB of RAM with a 20 MB hard disk - wow!) following Moore's Law nicely, and with C becoming the dominant programming language of the era, FORTRAN 77 - and hence See Why - was doomed. Witness was consequently re-written from scratch as a standalone product using C and the then new GUI technology (initially GEM, a version that was never released, then OS/2's Presentation Manager and finally Microsoft Windows, OSF/Motif and OpenWindows).

Consequently, from around 1987 onwards, C started playing a more significant role in my programming life. However, I always doubted whether C was a more efficient language than FORTRAN 77 - such was my indoctrination into the world of FORTRAN. (Since FORTRAN 77 was so limited, this issue was always a moot one for me.)

By the early 1990's, I regarded myself as a C programmer. I had heard rumours of C++, but - without ever studying or using it - dismissed it because I had heard that it was too high-level and took control away from the programmer. Sometime around 1992, I bought a copy of Borland C++ and Bjarne Stroustrup's book, The C++ Programming Language, Second Edition and was blown away! It was clear to me, as a simulation modeler, that C++ was a far superior language to C. It was also clear to me that C++ language features such as exception handling meant that it was possible to write code in C++ that was faster and safer than the equivalent C code (with explicit return value and/or errno checking in place of exception handling). However, most C programmers regard it as the more efficient language of the two, despite the fact that C is a sub-set of C++.

So, almost overnight, I became a C++ programmer.

By the mid-1990's, I had discovered Linux and the Perl programming language. Perl was a great tool/utility development language, but C++ surpassed it for application development and for simulations in particular.

However, as time went on, C++ seemed to stagnate. Whilst I loved the STL, the standard library was rather bare, with no standard GUI interface in particular and no standard support for writing multi-threaded applications either.

When the hype about Java started to do the rounds in the mid 1990's, I started to dabble with it. I liked the single source file approach (rather than the source-and-header file approach of C and C++), automated garbage collection, comprehensive standard library and its simplicity and elegance - but found it slow and, without multiple inheritance or support for generic programming, restrictive by comparison to C++.

When C# arrived on the scene, I was even more dubious. Although Microsoft marketed it at C and C++ developers, and named it as though a successor to the two, it was clear to me that it was Microsoft's embraced and extended version of Java. The biggest differences between the two languages (aside from ownership) were that C# dropped the checked exception feature of method declarations (a theme that Microsoft Visual C++ developers will also be familiar with) and introduced a number of interesting new features (attributes, reflection, events, delegates, properties, static constructors, etc. - some of which have since been added to Java).

Whilst I envied the simplicity and elegance of the two languages (compared to the clunkiness of C++), and the standard libraries that each had available, I disliked their proprietary status, slow execution speed and lack of multiple inheritance and generics. Then C# 2.0 came along, and I was impressed by the generics implementation. When I discovered that the Mono Project was supporting both C# and .NET as open source on Linux, it was enough for me to adopt it as the language for my long-time goal of developing an open-source simulation library.

I have always valued simplicity and clarity over performance, so - given the elegance of the language and its standard library - I was able to overlook C#/.NET's perceived run-time overhead (compared to C++) and its frustrating lack of multiple inheritance. However, simulations need to be high-performance and - as any simulation modeler will tell you - simulations cannot run fast enough! The faster a simulation runs, the faster you can spot bugs and fix them. The quicker simulation runs complete, the quicker you can get your reports completed. The less time it takes to perform one experiment, the more time you have to squeeze in another. Simulation run-speed is very important!

Politics then began to enter into the equation.  Firstly, Sun announced that they were to open-source Java and, shortly after that, came the now infamous agreement between Novell (sponsors of the Mono implementation of C#/.NET) and Microsoft. I was also experiencing a number of compiler bugs in the Mono C# compiler, and was less than impressed with the Mono team's handling of my bug reports.  When I considered these facts and combined them with a cursory examination of the goodies in Java 5 (generics, attributes, etc.), I decided to switch Facsimile development from C#/.NET to Java. I quickly became hugely disappointed in Java's generics implementation and hastily switched back again.

Of course, when you sup with the devil, or Microsoft, you need to use a very long spoon. Microsoft's abuse of the ISO/IEC voting procedures to fast-track it's appalling OOXML document "standard" on the world - in a desperate attempt to hold on to its monopoly of office software - woke me up to some harsh realities.  Also, co-incidentally, I found that my C# code was getting more and more complex because of the lack of support for multiple inheritance.  (One day, I'll try to find the time to write up exactly what these problems were.  It's currently unfashionable to like multiple inheritance - which has it's flaws, no doubt about that - but I've come to regard it as an indispensable feature of a modern programming language.)  So, I moved Facsimile to the completely unproprietary C++ language (thanks AT&T for having the wisdom to do that!).

So, now that I'm nearing the first release of Facsimile (honest!), I've started to become interested in language benchmarking. Many years ago, I bought a copy of Kernighan and Pike's excellent book, The Practice of Programming - if you write computer software of any type, whether you're a seasoned veteran or a novice, then I heartily recommend this book to you. In section 3.8, titled Performance, they compare the execution times of a text processing program in a variety of languages: C, Java, C++, Awk and Perl. Whilst it was no surprise that the C version ran the fastest, I was amazed to see that both Perl and Awk came in a close second! C++ came next (largely due to a bad STL deque implementation) with Java bringing up the rear. (It should be noted that Java was still a young language at the time the book was written, and that it's run-time performance has improved since.)

Whilst I was dubious about their C++ results, it struck me as interesting that two scripting languages could outperform a compiled language. Also, it seemed that Java ought to be able to execute faster than scripting languages, since it doesn't need to first parse the source code into an intermediate code (this is done by the Java compiler and is not performed at run-time). Given their similarity, Java and C# ought to be comparable, too.

An interesting paper by Lewis and Neuman even goes so far as to argue that Java is both theoretically and actually faster than C++. It is certainly a fact that memory usage plays an important role in execution speed, and that languages that use memory more effectively can run faster - even if they have more instructions to execute. (Fetching data from main memory is a significantly slower operation than fetching from the processor's on-board cache. Fetching paged virtual memory back from a disk is significantly slower still.)

The problems with language benchmarking are many:

  • The benchmark code needs to be written optimally in each language, taking advantage of each language's unique features and standard library. You cannot simply implement the benchmark as a common algorithm in each language.
  • The benchmark has to be relevant to the application you're interested in. For simulations, whilst graphical performance is important, raw performance (with graphics turned off or disabled) is far more so. Start-up speed (Java and C# are slower than C++ because of the just-in-time compilation of their bytecodes) is barely relevant at all.
  • Run-time bottlenecks related to the hardware and operating system software configuration need to be eliminated so that language performance can be assessed directly. (That said, if one language requires more resources than another, then that is most certainly of interest.)
  • I/O operations (it's tempting for benchmark applications to process stored data or write output) need to be handled with care, since they are often slow operations - confusing the relative performance of the benchmark code itself.

On a lighter note, there is an interesting language benchmark system hosted by Debian: The Computer Language Benchmarks Game.

For now, it appears that the jury is still out. I'll blog more on this in due course...