Integration Tests in Sonar

Integration tests are another important aspect of analyzing a project's overall health that Sonar does not yet support out of the box. To get this functionality, you'll need to build a couple of Sonar plugins (or try using the ones that I built) that will instrument your integration test code, run the integration tests, and collect the integration test results as well as the new coverage data.

It sounds like a tall order, but a lot of the work has already been done for you.

Sonar runs the unit tests in your project automatically by way of surefire. Sonar has a surefire plugin which executes the surefire:test goal and then collects the results by reading the TEST-xxx.XML files that it produces.

So, why not do the same with failsafe, the maven plugin for running integration tests? Sounds pretty simple and there are only a couple of catches.

The first catch is that there are not pre-defined metrics in Sonar for integration tests. So, if you don't mind piggybacking on the unit test metrics--meaning that your success, failure, and coverage numbers will be aggregated across both unit and integration--then simply copy the SurefireSensor, changing only the part where you are saving the metrics to updating existing metrics.

The second is a corner case. What if you don't have any unit tests, but you do have integration tests? If the surefire sensor doesn't find any unit tests to run, then it preemptively sets one of the metrics that later on is overwritten by the UnitTestDecorator (read: things explode). The only way around this one (that I've found) is to add another sensor to our failsafe plugin that, when the Sonar batch process first starts up, creates a dummy surefire empty test result for the surefire sensor to catch. While temporary and dummy files give me an icky feeling inside, it does the trick.

One more involved way around both of these is to create your own set of metrics and a decorator to display them. In this way, you wouldn't collide with any of Sonar's efforts to run unit tests. I haven't done this, yet, and I think that there might be more involved than that, so I didn't touch that route.

Okay, the failsafe plugin takes care of running and reporting on integration test execution. What about code coverage?

My coverage tool of choice is Emma. A while ago, Sonatype wrote an article explaining how to use emma4it to add integration test code coverage to emma. So, we can follow the same pattern, creating a Sonar plugin to execute the appropriate maven commands.

The only tricky part here was telling Sonar when to run each command. As far as I understand it, one cannot specify maven lifecycle points at which to run each maven goal. Instead, Sonar invokes all the goals serially at the point when it's that sensor's turn to execute.

What we really want is emma:instrument and emma4it:instrument-project-artifact to happen together, then the failsafe sensor, and then emma4it:report. Hmm...

The way we solved it was to have three different MavenPluginHandlers and Sensors in our emma4it sonar plugin. The emma:instrument and emma4it:instrument-project-artifact are dependent on emma finishing and emma4it:report is configured as running in the Phase.Name.POST phase. Failsafe, also specified as being dependent on code coverage finishing falls by default in between the two.

If there were a closer mapping to post-integration-test, process-classes, etc. in the future, this piece would become a lot cleaner.

I'm going to spend a bit of time cleaning up my code before I upload it, but after that, you are free to code by example. :)
Enhanced by Zemanta

Josh Cummings

"I love to teach, as a painter loves to paint, as a singer loves to sing, as a musician loves to play" - William Lyon Phelps

1 comment:

  1. Hi,
    Thanks for this very interesting entry.
    I'm very interested by having the "Test Success" part of sonar dashbord with my integration tests results.
    If I have correctly read your entry, you have developed a failsafe sonar plugin to do that. If it is the case, I'm very interested by your plugin. If you could say me where I can download it, I would be very happy !

    Of course, I understand if you say that this is not possible ;)

    ReplyDelete