Scala/test/benchmarks

Scala library benchmarks

This directory is used by the bench subproject of the Scala sbt build. It makes use of the sbt plugin for JMH.

About the benchmarks

Benchmarks are built with the reference compiler ("starr") using the library built from the library project ("quick"). If you want to test compiler changes you need to bootstrap a new compiler.

The benchmarking classes are organized under test/benchmarks/src/main/scala, in the same package hierarchy as the classes that they test.

The benchmarking classes use the same package hierarchy as the classes that they test in order to make it easy to expose members of the class under test in package-private scope, should that be necessary for benchmarking.

There are two types of classes in the source directory: those suffixed Benchmark, and a few that are suffixed Runner. (The latter are described below, under "Custom runners".)

Running a normal benchmark

Use bench/Jmh/run and provide the fully qualified name of the benchmark class:

bench/Jmh/run scala.collection.mutable.ListBufferBenchmark

Results are printed to standard output.

Custom runners

Some benchmarks have custom runners. A custom runner can be useful for setting appropriate JMH command options, and for processing the JMH results into files that can be read by other tools, such as Gnuplot.

Assuming that we're benchmarking scala.collection.mutable.OpenHashMap, the custom runner (if there is one) would likely be named scala.collection.mutable.OpenHashMapRunner. Using this example, one would run

bench/Jmh/runMain scala.collection.mutable.OpenHashMapRunner

in the Scala sbt build.

Custom runner results are written to ../../target/jmh-results/ (i.e. the main Scala build's target, not the one that contains the benchmark class files). jmh-results gets deleted on an sbt bench/clean, so you should copy these files out of target if you wish to preserve them.

If you want to make your own custom runner, extend the benchmark.JmhRunner trait, for the standard behavior that it provides. This includes creating output files in a subdirectory of target/jmh-results derived from the fully-qualified package name of the Runner class.

Some useful HotSpot options

Adding these to the Jmh/run or Jmh/runMain command line may help if you're using the HotSpot (Oracle, OpenJDK) compiler. They require prefixing with -jvmArgs. See the Java documentation for more options.

Viewing JIT compilation events

Adding -XX:+PrintCompilation shows when Java methods are being compiled or deoptimized. At the most basic level, these messages will tell you whether the code that you're measuring is still being tuned, so that you know whether you're running enough warm-up iterations. See Kris Mok's notes to interpret the output in detail.

Consider GC events

If you're not explicitly performing System.gc() calls outside of your benchmarking code, you should add the JVM option -verbose:gc to understand the effect that GCs may be having on your tests.

"Diagnostic" options

These require the -XX:+UnlockDiagnosticVMOptions JVM option.

Viewing inlining events

Add -XX:+PrintInlining.

Viewing the disassembled code

If you're running OpenJDK or Oracle JVM, you may need to install the disassembler library (hsdis-amd64.so for the amd64 architecture). In Debian, this is available in the libhsdis0-fcml package. For an Oracle (or other compatible) JVM not set up by your distribution, you may also need to copy or link the disassembler library to the jre/lib/architecture directory inside your JVM installation directory.

The JITWatch project has hsdis build instructions. One way to obtain HSDIS is to use the binaries which are used in the Graal build.

To show the assembly code corresponding to the code generated by the JIT compiler for specific methods, add -XX:CompileCommand=print,scala.collection.mutable.OpenHashMap::*, for example, to show all of the methods in the scala.collection.mutable.OpenHashMap class.

To show it for all methods, add -XX:+PrintAssembly. (This is usually excessive.)

Using JITWatch

JITWatch is useful to understand how the JVM has JIT-compiled code.

If you install hsdis, as described above, machine code disassembly is also created.

You can generate the hotspot.log file for a benchmark run by adding the required JVM options to JMH benchmark execution:

sbt:root> bench/Jmh/run scala.collection.mutable.ArrayOpsBenchmark.insertInteger -psize=1000 -f1 -jvmArgs -XX:+UnlockDiagnosticVMOptions -jvmArgs -XX:+TraceClassLoading -jvmArgs -XX:+LogCompilation -jvmArgs -XX:LogFile=target/hotspot.log -jvmArgs -XX:+PrintAssembly
...
[info] Loaded disassembler from /Users/jz/.jabba/jdk/1.8.172/Contents/Home/jre/lib/hsdis-amd64.dylib
[info] Decoding compiled method 0x0000000113f60bd0:
[info] Code:
[info] [Disassembling for mach='i386:x86-64']
[info] [Entry Point]
[info] [Constants]
[info]   # {method} {0x000000010ffa0000} 'hashCode' '()I' in 'java/lang/String'
[info]   #           [sp+0x40]  (sp of caller)
[info]   0x0000000113f60d40: mov    r10d,DWORD PTR [rsi+0x8]
[info]   0x0000000113f60d44: shl    r10,0x3
...
[info] # Run complete. Total time: 00:00:30
[info] Benchmark                        (size)  Mode  Cnt       Score      Error  Units
[info] ArrayOpsBenchmark.insertInteger    1000  avgt   10  188199.582 ± 5930.520  ns/op

JITWatch requires configuration of the class and source path. We generate that with a custom task in our build:

sbt> bench/Jmh/jitwatchConfigFile
...
jmh
...
[info] ^-- UNRESOLVED DEPENDENCIES warnings above are normal, please ignore
[info] After cloning https://github.com/AdoptOpenJDK/jitwatch to $JITWATCH_HOME, compile and launch with:
[info] mvn -f $JITWATCH_HOME clean compile exec:java -Djitwatch.config.file=/Users/jz/code/scala/test/benchmarks/target/jitwatch-compile.properties -Djitwatch.logfile=/Users/jz/code/scala/test/benchmarks/target/hotspot.log
[info] Note: Add, for example, `-Djitwatch.focus.member="scala/collection/mutable/ArrayOpsBenchmark insertInteger (Lorg/openjdk/jmh/infra/Blackhole;)V"` to focus UI on a method of interest.
sbt> ^C

Follow instructions in the output above and start gleaning insights!

Useful reading

• OpenJDK advice on microbenchmarks
• Brian Goetz's "Java theory and practice" articles:
  • "Dynamic compilation and performance measurement"
  • "Anatomy of a flawed benchmark"
• Doug Lea's JSR 166 benchmarks
• "Measuring performance" of Scala parallel collections