Curious JMH results between Java 8 and Java 9

2:17 PM 0 Comments

I've recently been playing around with JMH and doing some comparisons between Java 8 and Java 9. I wrote the following toy benchmark, learning from the example that Julian Ponge wrote up in his article Avoiding Benchmarking Pitfalls on the JVM. This is my simple attempt to apply the principle:



import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Scope;

@State(Scope.Benchmark)
@Fork(1)
public class BenchmarkComparison {
  public double sqrt(double what) {
    return Math.exp(0.5 * Math.log(what));
  }

  private double what = 10d;

  @Benchmark
  public double baseline() {
    return Math.sqrt(what);
  }

  @Benchmark    
  public double correct() {
    return sqrt(what);
  }

  @Benchmark
  public double constantFolding() {
    return sqrt(10d);
  }

  @Benchmark
  public void deadCodeElimination() {
    sqrt(what);
  }

  @Benchmark
  public void deadCodeAndFolding() {
    sqrt(10d);
  }
}


Julian's post is intended to demonstrate common pitfalls that Java engineers fall into when it comes to benchmarking, with three of the methods above indicating incorrect ways to create a benchmark. I invite you to read his informative post to get more background information, if you like.

Running the following JMH benchmark in Java 8, I get the following results:


And here are the results in Java 9 on the same machine:


While this is a great example for why benchmarks need to be run on consistent JVM versions, what interests me more is why are the results in Java 9 are so much "smoother"? Why are they even the same order of magnitude?

I don't have the example handy, but I had a similar experience with Julian's very first experiment, with running several benchmarks in the same JVM run, which is a "no no". In Java 8, I saw the same behavior as Julian, but in Java 9, I didn't until I added a third test to the benchmark. If I only added two, I didn't see the dramatic performance degredation.

Any ideas?


Josh Cummings

"I love to teach, as a painter loves to paint, as a singer loves to sing, as a musician loves to play" - William Lyon Phelps

0 comments: