Micro-Benchmarking Scala Code - The Easy Way

Posted by Mathias in [scala]

19 Apr 2011

While evaluating, which of the numerous available Scala JSON implementations to use for spray once again I found myself in need of some tool simplifying the writing of proper JVM benchmarks.
Manually writing benchmarks for the JVM that actually measure what you intend to measure is much harder than it initially appears. There are quite a few rules to keep in mind, even making your hand-written benchmark comply with just half of them requires a lot of attention.

As ever so often other people have already had the same problem and decided to do something about it. In this case a few guys at Google came up an open-source framework for JVM micro-benchmarks called Caliper. Caliper takes care of many of the hard and complex parts of a good micro-benchmark, like properly “warming up” the JVM before actually taking measurements or determining how often to run your code in order to achieve a good trade-off between measuring accuracy and total runtime.
Of course there are still a few things you have to watch out for, even when using Caliper. However, the framework can be of great help when you’d like to decide between two alternative implementations of the same algorithm based on performance, determine whether a particular “optimization” is really worth the effort or find out how a certain JVM setting affects the runtime performance of some ingenious code of yours.

In order to make Caliper easily accessible to Scala developers I whipped up a small SBT template project that you can use as the basis for your own benchmarking endeavours. As a simple example the project already contains a small benchmark testing the performance of ”foreaching” over a Scala Array against a simple while loop as well as a @specialized custom for loop replacement implementation.
This is what the Caliper output looks like when run from within SBT on my machine:

mathias@cox ~/Documents/scala-benchmarking-template > sbt
[info] Building project scala-benchmarking-template 1.0.0-SNAPSHOT against Scala 2.8.1
[info]    using Project with sbt 0.7.5 and Scala 2.7.7
> run 
[info] 
[info] == copy-resources ==
[info] == copy-resources ==
[info] 
[info] == compile ==
[info]   Source analysis: 3 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling main sources...
[info] Compilation successful.
[info]   Post-analysis: 12 classes.
[info] == compile ==
[info] 
[info] == run ==
[info]  0% Scenario{vm=java, trial=0, benchmark=Foreach, length=10} 24.49 ns; ?=0.23 ns @ 7 trials
[info]  8% Scenario{vm=java, trial=0, benchmark=TFor, length=10} 7.41 ns; ?=0.07 ns @ 3 trials
[info] 17% Scenario{vm=java, trial=0, benchmark=While, length=10} 5.70 ns; ?=0.05 ns @ 3 trials
[info] 25% Scenario{vm=java, trial=0, benchmark=Foreach, length=100} 227.17 ns; ?=1.69 ns @ 3 trials
[info] 33% Scenario{vm=java, trial=0, benchmark=TFor, length=100} 40.27 ns; ?=0.13 ns @ 3 trials
[info] 42% Scenario{vm=java, trial=0, benchmark=While, length=100} 37.69 ns; ?=0.23 ns @ 3 trials
[info] 50% Scenario{vm=java, trial=0, benchmark=Foreach, length=1000} 5781.09 ns; ?=24.06 ns @ 3 trials
[info] 58% Scenario{vm=java, trial=0, benchmark=TFor, length=1000} 360.72 ns; ?=3.62 ns @ 5 trials
[info] 67% Scenario{vm=java, trial=0, benchmark=While, length=1000} 359.78 ns; ?=3.70 ns @ 10 trials
[info] 75% Scenario{vm=java, trial=0, benchmark=Foreach, length=10000} 58255.01 ns; ?=146.74 ns @ 3 trials
[info] 83% Scenario{vm=java, trial=0, benchmark=TFor, length=10000} 3838.16 ns; ?=24.80 ns @ 3 trials
[info] 92% Scenario{vm=java, trial=0, benchmark=While, length=10000} 3836.80 ns; ?=14.47 ns @ 3 trials
[info] 
[info] benchmark length       ns linear runtime
[info]   Foreach     10    24.49 =
[info]   Foreach    100   227.17 =
[info]   Foreach   1000  5781.09 ==
[info]   Foreach  10000 58255.01 ==============================
[info]      TFor     10     7.41 =
[info]      TFor    100    40.27 =
[info]      TFor   1000   360.72 =
[info]      TFor  10000  3838.16 =
[info]     While     10     5.70 =
[info]     While    100    37.69 =
[info]     While   1000   359.78 =
[info]     While  10000  3836.80 =
[info] 
[info] vm: java
[info] trial: 0
[info] == run ==
[success] Successful.
[info] 
[info] Total time: 113 s, completed Apr 19, 2011 10:18:32 PM
> _

In order to create your own benchmark all you have to do is replace the respectively marked code snippets with your own.

Let me know, how Caliper works for you…

Cheers,
Mathias

View Comments