Maven, SonarQube, JaCoCo, Lombok and TeamCity

Posted on October 29, 2017 Updated on October 29, 2017

As part of striving for better, I’ve added a build step in TeamCity to the tcWebHooks build to analyse my code in SonarQube. This has revealed a large amount of debt (30 days) to tidy up, and I plan to tackle that as I work on the code base. You can see a screenshot of the analysis by SonarQube below.

During this exercise I realised that a number of the tools required were already supported by TeamCity with built in features or external plugins, and I didn’t have to make changes to my build (pom.xml). I understand that this is arguably not a great solution if you adhere to the configuration as code philosophy, because the Sonar configuration – and the steps to invoke it – now live outside of the source code I have in git.

For my webhooks plugin I have full control of the process from coding to Continuous Integration (CI) to automated deployment in test (CD) and then to doing a release on GiHub. However, I have worked in environments where the CI team is not necessarily the same as the development team. In one of my roles, when we were first moving to a CI model and encouraging as many development teams as possible to try CI, I was that guy. I owned the server and its configuration, but had no commit rights on any of the hundreds of builds my server ran.

That is why I can understand both sides of the argument for putting sonar configuration in the project POM in git, versus in the TeamCity build configuration – which can also be in source control.

One other consideration, is that I have the build running on two build servers. One runs at home on a local TeamCity instance, and one runs on the Jetbrains TeamCity instance which they generously make available for qualifying open-source projects.

My local build runs Sonar analysis on every local commit. This may seem like a waste, but I am coding on my own time, and therefore want rapid feedback when I am coding, not on some nightly build, which I may not get a chance to review for another week. Running this build on own TeamCity instance lets me interact with my own Sonar server (running in a docker instance) and also allows me to install any plugins I want, which I obviously can’t do with the Jetbrains TeamCity instance.

Adding JaCoCo test coverage analyser

JaCoCo is easy to enable in TeamCity. My build uses Maven, so I go into the the build settings, and edit the Maven build step. Under Code Coverage, choose JaCoCo as the coverage runner and add the classfile directories. My build is a maven multiple project build, so am I using ** at the start to let JaCoCo find all the classes from all the sub-projects.

Setting up a SonarQube instance

For me, I just run SonarQube using docker-compose, based on a clone of harbur/docker-sonarqube. I did this a while ago, so there is probably a better way now.

Running Sonar as part of your build

In TeamCity, Sonar support can be added by installing the SonarQube plugin. This plugin was developed by a TeamCity team member, and allows the configuration of SonarQube servers at any project level (eg, _Root or below that), which are then available to submit analysis to. See the docs for the plugin for info on adding your SonarQube server to TeamCity.

The SonarQube plugin also adds a new Build Runner, so invoking the Sonar scanner is configured as a build step on your build configuration in TeamCity.

Here are the settings from my Sonar build step.

• Project name
  I am using a name based on the the build name and branch. This means that each build is in Sonar is at the top level as a “project”. It works for me because I only have a handful of builds. Using the branch name means that builds from different branches have their own analysis history. I have a long running branch for the next iteration (which just migrated to master), and a previous release branch which gets maintenance fixes.
• Project key
  This uses the same logic as above, but using the id since that won’t contain weird characters.
• Project version
  Sonar rolls builds of the same version together, so using the maven version allows me to see progress from one release to another, rather than from one build to another. The maven version is typically something like “1.1-alpha11.snapshot“.
  I recently changed this from using the TeamCity build number. SonarQube indicates how you are doing compared with the previous version. Using the build number as the version changes the version for every build, so a commit that has some bad stuff makes the analysis go Red, and then the next build it goes Green again which is somewhat misleading. You can see this in the history on the right of the screenshot at the top of this post.
  Because I recently changed this, the screenshot above shows “since 170” as that was the build number when I changed that setting.
• Sources, Tests and Binaries
  These are the standard maven locations for sources and test, but don’t forget to set the binaries location too. This was important for sending the instrumented code to Sonar, otherwise your coverage does not show up in SonarQube.
• Modules
  These are the sub-modules within my maven build. I would love to not have to specify these here somehow, because if I add a new module in a branch and update this field, sonar fails the build for branches that don’t have the module present. This is a case where having the config in git with the branch would solve this problem.
• Additional parameters
  This is the magic setting to get annotation generated code analysed properly. Before I added this, I was getting complaints from SonarQube like “Unused private fields should be removed”, even though Lombok has generated getters and setters for them. For this to work, you also need to ask maven to copy the libraries here so that sonar can find them (see next section).

Telling Sonar about Lombok, @NotNull and other annotations

During sonar analysis, I was getting errors similar to:

Class 'abc.xyz.Classname' is not accessible through the ClassLoader.

and Sonar would add failures with “Unused private fields should be removed”. My presumption is that the code used to generate these methods was not available to Sonar at analysis time.
There are two changes I made to resolve this:

1. Tell maven to copy its dependencies to somewhere Sonar can find them. This is simple as adding

   dependency:copy-dependencies

   to the goals that maven will execute, and is configured in the maven build step in TeamCity.

2. Add the following to the sonar build step “Additional parameters” (see above section).

   -Dsonar.java.libraries=target/dependency/*.jar

Then run your build and watch the good (or bad) news appear in SonarQube.

Lombok and Code Coverage

I love Lombok for reducing boiler plate code. Simply adding @Data to a class adds @ToString, @EqualsAndHashCode, @Getter, @Setter and @RequiredArgsConstructor. With with five extra characters, Lombok adds a lot of new code to my classes but keeps them clean to read.

However, unless I have tests for all this code, it actually drives the coverage down. Sometimes I end up writing extra tests to test Lombok’s code which seems absurd. Other times I replace @Data with the handful of annotations I really need. Some classes need getters on most fields, but only setters on some. After fixing those up, I end up with classes that now have twice as many lines as when I added just the one @Data annotation.

I’ve not really come to a good middle ground on that one. Coverage or clean code. It’s a trade off.

In conclusion

I’ve found SonarQube to be a good spotlight on where my code needs addressing, and have learnt a lot about Java and Java development best practices (according to the “SonarWay”). I was making really good progress on cleaning up some debt and had the bug count down to one! Then I figured out how to fix the missing annotation processing code, and it found 12 new bugs and 91 new code smells. I will endeavour to keep chipping away at it until all the bugs and vulnerabilities are down to zero, and hopefully some debt paid down.

Apart from the coverage dilemma above, SonarQube has been a positive experience. It’s a good way to learn better practices, and be able to chart my progress doing it.