Sonar tests coverage

Maven, failsafe, sonar and jacoco are in a boat …

Yesterday I wanted to setup the code coverage in our sonar instance for integration tests launched using the failsafe plugin of Apache Maven. The awaited result in sonar is something like this :

Sonar tests coverage
Sonar tests coverage


I found several constraints while implementing it :

  • Integration tests may require to have some tasks called in the pre-integration-test phase (for example to generate some random ports to launch an application server and avoid to conflicts on your CI server)
  • Sonar can itself execute your unit tests managed by maven and its surefire plugin but it has nothing for integration tests by default.
  • Sonar can call a maven lifecycle or mojo (property sonar.phase) before being executed but we cannot use this to call integration tests without having it re-executing a second time your unit tests.
  • Sonar cannot display integration tests results (only the coverage)

Digging into various posts (see references) I found a satisfying (but not perfect) solution to fit my needs.

In my corporate POM I added these properties

<!-- ************************ -->
<!-- Sonar/Reporting settings -->
<!-- ************************ -->
<!-- Sonar/Jacoco integration. Note that these properties need to be defined outside the "coverage" profile
because we want to be to able to execute mvn sonar:sonar without passing a profile -->
<!-- Tells Sonar to use jacoco for coverage results -->
<sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
<!-- Jacoco version to use -->
<jacoco.version>0.6.3.201306030806</jacoco.version>
<!-- The Sonar Jacoco Listener for JUnit to extract coverage details per test -->
<sonar-jacoco-listeners.version>1.4</sonar-jacoco-listeners.version>
<!-- Don't let Sonar execute tests. We will ask it to Maven -->
<sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
<!-- The system property jacoco.outputDir needs to be override on the command line
with an absolute path if you want to merge results from all modules.
Example in a Jenkisn build where ${WORKSPACE} is defined and your project in the root directory of the workspace :
mvn clean install -Prun-its,coverage -Djacoco.outputDir=${WORKSPACE}/target
Note that unfortunately using the following does not work because of

http://jira.codehaus.org/browse/SONAR-3427:

<jacoco.outputDir>${session.executionRootDirectory}/target/</jacoco.outputDir>
-->
<jacoco.outputDir>${project.build.directory}</jacoco.outputDir>
<!-- Jacoco output file for UTs -->
<jacoco.out.ut.file>jacoco-ut.exec</jacoco.out.ut.file>
<!-- Tells Sonar where the Jacoco coverage result file is located for Unit Tests -->
<sonar.jacoco.reportPath>${jacoco.outputDir}/${jacoco.out.ut.file}</sonar.jacoco.reportPath>
<!-- Jacoco output file for ITs -->
<jacoco.out.it.file>jacoco-it.exec</jacoco.out.it.file>
<!-- Tells Sonar where the Jacoco coverage result file is located for Integration Tests -->
<sonar.jacoco.itReportPath>${jacoco.outputDir}/${jacoco.out.it.file}</sonar.jacoco.itReportPath>

And I added this new profile

<profile>
  <id>coverage</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <argLine>${jacoco.agent.ut.arg}</argLine>
          <!-- Specific to generate mapping between tests and covered code -->
          <properties>
            <property>
              <name>listener</name>
              <value>org.sonar.java.jacoco.JUnitListener</value>
            </property>
          </properties>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <configuration>
          <argLine>-Xmx1024m -XX:MaxPermSize=256m ${jacoco.agent.it.arg}</argLine>
          <!-- Specific to generate mapping between tests and covered code -->
          <properties>
            <property>
              <name>listener</name>
              <value>org.sonar.java.jacoco.JUnitListener</value>
            </property>
          </properties>
          <!-- Let's put failsafe reports with surefire to have access to tests failures/success reports in sonar -->
          <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>${jacoco.version}</version>
        <executions>
          <!-- Prepares a variable, jacoco.agent.ut.arg, that contains the info to be passed to the JVM hosting the code
being tested. -->
          <execution>
            <id>prepare-ut-agent</id>
            <phase>process-test-classes</phase>
            <goals>
              <goal>prepare-agent</goal>
            </goals>
            <configuration>
              <destFile>${sonar.jacoco.reportPath}</destFile>
              <propertyName>jacoco.agent.ut.arg</propertyName>
              <append>true</append>
            </configuration>
          </execution>
          <!-- Prepares a variable, jacoco.agent.it.arg, that contains the info to be passed to the JVM hosting the code
being tested. -->
          <execution>
            <id>prepare-it-agent</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>prepare-agent</goal>
            </goals>
            <configuration>
              <destFile>${sonar.jacoco.itReportPath}</destFile>
              <propertyName>jacoco.agent.it.arg</propertyName>
              <append>true</append>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  <dependencies>
    <dependency>
      <groupId>org.codehaus.sonar-plugins.java</groupId>
      <artifactId>sonar-jacoco-listeners</artifactId>
      <version>${sonar-jacoco-listeners.version}</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</profile>

I already had one to launch integration tests with failsafe

<!--
This profile is used to launch integration tests with the failsafe plugin

http://maven.apache.org/plugins/maven-failsafe-plugin/

-->
<profile>
  <id>run-its</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <executions>
          <execution>
            <id>integration-test</id>
            <phase>integration-test</phase>
            <goals>
              <goal>integration-test</goal>
            </goals>
          </execution>
          <execution>
            <id>verify</id>
            <phase>verify</phase>
            <goals>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</profile>

Thus in my CI server (Jenkins) I just have to create a job that calls

mvn verify -Prun-its,coverage -Djacoco.outputDir=${WORKSPACE}/target

And I add Sonar as a post build step which will grab tests and coverage results.

Note : -Djacoco.outputDir=${WORKSPACE}/target is necessary only if you want to merge all coverage results across several modules.

With my test project I obtain the expected result.

References

  1. Code Coverage by Integration Tests for Java Project
  2. Code Coverage by Unit Tests for Java Project
  3. Sonar Analysis Parameters
  4. Sonar examples
  5. Code coverage configuration for Xwiki (thanks @vmassol)
  6. SONAR-3427 : Setting sonar.jacoco.itReportPath property via maven ${session.executionRootDirectory} does not work : I agree with Fabrice I didn’t know myself this property but it is sad to not implement it as it is working in some others plugins (the question is to know if there is a minimum version of Maven to support it)
  7. SONARJAVA-94 : Provide JUnit Listener to record coverage per test
  8. Tracking Integration Test Coverage with Maven and SonarQube
  9. Maven Sonar plugin execute surefire tests twice when setting the property sonar.phase=post-integration-test
  10. Failsafe tests results in sonar
  11. Separating Integration and Unit Tests with Maven, Sonar, Failsafe, and JaCoCo
GD Star Rating
loading...
GD Star Rating
loading...
Maven, failsafe, sonar and jacoco are in a boat ..., 10.0 out of 10 based on 4 ratings

8 thoughts on “Maven, failsafe, sonar and jacoco are in a boat …”

  1. Nice article Arnaud. Just a typo about your CI server name : Jenkins instead of Kenkins

    GD Star Rating
    loading...
    GD Star Rating
    loading...
  2. Hi,

    thanks for your article. Can you explain what does:

    listener
    org.sonar.java.jacoco.JUnitListener

    do? It does not seem to make a difference for me if I incluce it or not. Or maybe I just do not see the difference.

    Thanks,
    Ronald

    GD Star Rating
    loading...
    GD Star Rating
    loading...
  3. Great Article! 10/10 as this was the only place I was able to reference to get the proper reporting working in Sonar. I have one side effect I’m wondering by chance that you have any insight into – I can no longer “Test File” nor “Run Focused Test Method” in Netbeans IDE 7.4. This runs fine from Jenkins as an automated job.

    The error observed is:

    Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-cli) on project test-project: Execution default-cli of goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test failed: There was an error in the forked process
    org.apache.maven.surefire.util.SurefireReflectionException: org.sonar.java.jacoco.JacocoController$JacocoControllerError: Unable to access JaCoCo Agent - make sure that you use JaCoCo and version not lower than 0.6.2.; nested exception is org.sonar.java.jacoco.JacocoController$JacocoControllerError: Unable to access JaCoCo Agent - make sure that you use JaCoCo and version not lower than 0.6.2.
    org.sonar.java.jacoco.JacocoController$JacocoControllerError: Unable to access JaCoCo Agent - make sure that you use JaCoCo and version not lower than 0.6.2.

    GD Star Rating
    loading...
    GD Star Rating
    loading...
  4. Wonderful work !
    It spared me hours!
    Your page should be included in the official SonarQube documentation.
    Thanks again

    GD Star Rating
    loading...
    GD Star Rating
    loading...
  5. Hello

    Thanks for this good reference.

    Your solutions sounds good to apply on my project, but I’m getting a org.sonar.api.test.exception.CoverageAlreadyExistsException, which points to a test method and class that this test covers. Have you faced some related problem? Seems that it happens because a class is covered more than one test.

    The reference for the class that throws the exception is: https://github.com/SonarSource/sonarqube/blob/master/sonar-core/src/main/java/org/sonar/core/test/DefaultTestCase.java

    GD Star Rating
    loading...
    GD Star Rating
    loading...
  6. I followed your example and got the following error.

    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.13:integration-test (default-cli) on project fms-arquilli
    an-tests: Execution default-cli of goal org.apache.maven.plugins:maven-failsafe-plugin:2.13:integration-test failed: There was an error i
    n the forked process
    [ERROR] org.apache.maven.surefire.util.SurefireReflectionException: org.sonar.java.jacoco.JacocoController$JacocoControllerError: Unable
    to access JaCoCo Agent – make sure that you use JaCoCo and version not lower than 0.6.2.; nested exception is org.sonar.java.jacoco.Jacoc
    oController$JacocoControllerError: Unable to access JaCoCo Agent – make sure that you use JaCoCo and version not lower than 0.6.2.
    [ERROR] org.sonar.java.jacoco.JacocoController$JacocoControllerError: Unable to access JaCoCo Agent – make sure that you use JaCoCo and v
    ersion not lower than 0.6.2.
    [ERROR] at org.sonar.java.jacoco.JacocoController.(JacocoController.java:48)

    My jacoco version is 0.7.2.201409121644

    Thank you.

    GD Star Rating
    loading...
    GD Star Rating
    loading...

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>