PNetMark FAQ

1. What is PNetMark?

3. How do I build PNetMark?

Normally you will unpack and build PNetMark as follows:

$ gunzip pnetmark-VERSION.tar.gz | tar xvf -
$ cd pnetmark-VERSION
$ ./configure
$ make

This assumes that you already have Portable.NET installed on your system. If you did not do a "make install" on Portable.NET, then you can specify the location of the Portable.NET build trees as follows:

$ ./configure --with-pnet=../pnet-0.5.0 --with-pnetlib=../pnetlib-0.5.0

A number of configure options can be supplied to change the C# compiler and Common Language Runtime that are used to build and run the benchmarks:

--with-cscc

Use Portable.NET's C# compiler, cscc.

--with-csc

Use Microsoft's C# compiler, csc.

--with-mcs

Use Mono's C# compiler, mcs.

--with-ilrun

Use Portable.NET's runtime engine, ilrun.

--with-ms

Use Microsoft's commercial runtime engine.

--with-clix

Use Rotor's runtime engine, clix.

--with-mono

Use Mono's JIT-based runtime engine, mono.

--with-mint

Use Mono's interpreter-based runtime engine, mint.

If you supply a compiler option, but not a runtime engine option, then configure will choose the runtime engine that corresponds to the compiler (e.g. --with-cscc and --with-ilrun). The same applies if you supply a runtime engine option, but not a compiler option.

If you supply neither a compiler option nor a runtime engine option, then configure will use the first of the Portable.NET, Mono, or Microsoft tools (in that order), depending upon what is installed on your system.

$ make check

$ ilrun pnetmark.exe

5. What do the benchmarks indicate?

Never believe benchmark numbers for engines that you cannot inspect the source code for. Vendors have been known to modify their engines specifically to get high scores on benchmarks. If you cannot see what hacks they've added to the code to lie to the benchmark, then you shouldn't believe the numbers.

For example, some JIT's get a disproportionately large value for loop benchmarks, but that is probably due to the JIT optimising the entire benchmark away, or by aggressively unrolling it. Real world applications don't normally benefit from such optimizations.

A JIT that lies to a benchmark may appear to be faster, but in fact will be slower. Lying JIT's spend extra time checking for optimisations that will not be needed in real applications. This extra checking slows the JIT down, and so real applications run slower than on an honest engine.

You must be careful with comparions between engines from different vendors. Different implementation techniques lead to different trade-offs. Benchmarks don't always accurately measure these trade-offs.

It is sometimes possible for an interpreter to out-perform a JIT if the application is I/O bound, and the interpreter has been optimised for I/O operations. A JIT that can perform fantastic loop optimisations will be useless on such an application.

As a general guide, when comparing the performance of dissimilar CLR's, make sure that you compile the binaries with the same compiler and optimization flags. For example, compile with cscc and then run the same binary with ilrun and mono. You want to measure the optimizations in the engine, not the optimizations in the compiler.

Finally, remember that most applications spend the bulk of their time waiting for the user to press a key or move the mouse, or waiting for a remote process to respond to a request. No amount of fancy optimisations can speed up the user or the remote machine. Benchmarks are a guide to performance, but never an absolute indicator.

8. How do I add a new benchmark?

New benchmarks in pnetmark.exe are provided by classes that implement the "IBenchmark" interface. The following methods and properties must be supplied:

Initialize

Initialize the benchmark. This is called once per benchmark.

Name

Name of the benchmark, for reporting purposes.

Magnification

An integer value that is used to scale the number of runs to produce a usable score value.

Run

Run the benchmark once. This will be called repeatedly until the caller is satisified that it has collected sufficient information.

CleanUp

Clean up any temporary data that was used by the benchmark. This is called once per benchmark.

Run(new FooBenchmark());

9. What is the theory underlying the benchmarks?

We are deliberately avoiding this question of benchmark "validity". Instead, we have tried to faithfully reproduce well-known benchmarks that have been used elsewhere and are generally well understood (the one exception is the use of C#'s 2D array facility in the "Float" benchmark - the CaffeineMark uses nested 1D arrays).

For example, the SciMark and Linpack benchmarks are a direct port to C# of the Java versions. Other than syntactic differences, the code is identical. Questions as to benchmarking theory are best sent to the original authors of those benchmarks.

You are welcome to submit patches if you detect a discrepancy in our porting efforts.