TeamCity

A better way of integrating TeamCity with Slack.com using WebHooks

Posted on Updated on

I wrote a blog post in 2014 about how it’s possible to get tcWebHooks to send messages to slack. It turned out to be one of the more popular posts.

The good news is that in version 1.1 of tcWebHooks, it got whole lot easier.

TL;DR

  1. Install the latest tcWebHooks 1.1 (or higher) plugin into TeamCity.
  2. Create an incoming webhook in slack.
  3. Create a webhook in TeamCity and choose from one of the Slack payload formats
  4. That’s it!

 

Bundled Slack WebHook Templates

As of the time of writing (December 2017), there are two Slack specific webhook templates bundled with a tcWebHooks release.

1. The “Slack.com JSON templates” payload format has templates which produce a message in slack with some information about the build.

33641251-a070628e-da99-11e7-8060-a454b535d52a

2. The “Slack.com Compact Notification” has templates which produce a more compact notification.

33641310-da06b0c0-da99-11e7-9d2f-f8a30ecb6364

If neither of those are right for you, it’s possible to create your own template by either copying an existing one, or creating a new one.

WebHook Templates

A WebHook Template is a predefined payload that can be reused with multiple webhook configurations. I’ve prepared a few to get you started, but it’s very easy to modify them or create your own. The ones bundled with tcWebHooks include:

Creating or editing templates is accomplished using a WebUI built on top of the tcWebHooks REST API.  Further information about WebHook Templates is on the wiki.

 

Advertisements

Maven, SonarQube, JaCoCo, Lombok and TeamCity

Posted on Updated on

As part of striving for better, I’ve added a build step in TeamCity to the tcWebHooks build to analyse my code in SonarQube. This has revealed a large amount of debt (30 days) to tidy up, and I plan to tackle that as I work on the code base. You can see a screenshot of the analysis by SonarQube below.

sonarqube_results_screenshot.png

During this exercise I realised that a number of the tools required were already supported by TeamCity with built in features or external plugins, and I didn’t have to make changes to my build (pom.xml). I understand that this is arguably not a great solution if you adhere to the configuration as code philosophy, because the Sonar configuration – and the steps to invoke it – now live outside of the source code I have in git.

For my webhooks plugin I have full control of the process from coding to Continuous Integration (CI) to automated deployment in test (CD) and then to doing a release on GiHub. However, I have worked in environments where the CI team is not necessarily the same as the development team. In one of my roles, when we were first moving to a CI model and encouraging as many development teams as possible to try CI, I was that guy. I owned the server and its configuration, but had no commit rights on any of the hundreds of builds my server ran.

That is why I can understand both sides of the argument for putting sonar configuration in the project POM in git, versus in the TeamCity build configuration – which can also be in source control.

One other consideration, is that I have the build running on two build servers. One runs at home on a local TeamCity instance, and one runs on the Jetbrains TeamCity instance which they generously make available for qualifying open-source projects.

My local build runs Sonar analysis on every local commit. This may seem like a waste, but I am coding on my own time, and therefore want rapid feedback when I am coding, not on some nightly build, which I may not get a chance to review for another week. Running this build on own TeamCity instance lets me interact with my own Sonar server (running in a docker instance) and also allows me to install any plugins I want, which I obviously can’t do with the Jetbrains TeamCity instance.

Adding JaCoCo test coverage analyser

JaCoCo is easy to enable in TeamCity. My build uses Maven, so I go into the the build settings, and edit the Maven build step. Under Code Coverage, choose JaCoCo as the coverage runner and add the classfile directories. My build is a maven multiple project build, so am I using ** at the start to let JaCoCo find all the classes from all the sub-projects.

teamcity_maven_coverage_screenshot.png

Setting up a SonarQube instance

For me, I just run SonarQube using docker-compose, based on a clone of harbur/docker-sonarqube. I did this a while ago, so there is probably a better way now.

Running Sonar as part of your build

In TeamCity, Sonar support can be added by installing the SonarQube plugin. This plugin was developed by a TeamCity team member, and allows the configuration of SonarQube servers at any project level (eg, _Root or below that), which are then available to submit analysis to. See the docs for the plugin for info on adding your SonarQube server to TeamCity.

The SonarQube plugin also adds a new Build Runner, so invoking the Sonar scanner is configured as a build step on your build configuration in TeamCity.

Here are the settings from my Sonar build step.

teamcity_sonarqube_runner_screenshot.png

  • Project name
    I am using a name based on the the build name and branch. This means that each build is in Sonar is at the top level as a “project”. It works for me because I only have a handful of builds. Using the branch name means that builds from different branches have their own analysis history. I have a long running branch for the next iteration (which just migrated to master), and a previous release branch which gets maintenance fixes.
  • Project key
    This uses the same logic as above, but using the id since that won’t contain weird characters.
  • Project version
    Sonar rolls builds of the same version together, so using the maven version allows me to see progress from one release to another, rather than from one build to another. The maven version is typically something like “1.1-alpha11.snapshot“.
    I recently changed this from using the TeamCity build number. SonarQube indicates how you are doing compared with the previous version. Using the build number as the version changes the version for every build, so a commit that has some bad stuff makes the analysis go Red, and then the next build it goes Green again which is somewhat misleading. You can see this in the history on the right of the screenshot at the top of this post.
    Because I recently changed this, the screenshot above shows “since 170” as that was the build number when I changed that setting.
  • Sources, Tests and Binaries
    These are the standard maven locations for sources and test, but don’t forget to set the binaries location too. This was important for sending the instrumented code to Sonar, otherwise your coverage does not show up in SonarQube.
  • Modules
    These are the sub-modules within my maven build. I would love to not have to specify these here somehow, because if I add a new module in a branch and update this field, sonar fails the build for branches that don’t have the module present. This is a case where having the config in git with the branch would solve this problem.
  • Additional parameters
    This is the magic setting to get annotation generated code analysed properly. Before I added this, I was getting complaints from SonarQube like “Unused private fields should be removed”, even though Lombok has generated getters and setters for them. For this to work, you also need to ask maven to copy the libraries here so that sonar can find them (see next section).

Telling Sonar about Lombok, @NotNull and other annotations

During sonar analysis, I was getting errors similar to:

Class 'abc.xyz.Classname' is not accessible through the ClassLoader.

and Sonar would add failures with “Unused private fields should be removed”. My presumption is that the code used to generate these methods was not available to Sonar at analysis time.
There are two changes I made to resolve this:

  1. Tell maven to copy its dependencies to somewhere Sonar can find them. This is simple as adding
    dependency:copy-dependencies

    to the goals that maven will execute, and is configured in the maven build step in TeamCity.

  2. Add the following to the sonar build step “Additional parameters” (see above section).
    -Dsonar.java.libraries=target/dependency/*.jar

Then run your build and watch the good (or bad) news appear in SonarQube.

Lombok and Code Coverage

I love Lombok for reducing boiler plate code. Simply adding @Data to a class adds @ToString, @EqualsAndHashCode, @Getter, @Setter and @RequiredArgsConstructor. With with five extra characters, Lombok adds a lot of new code to my classes but keeps them clean to read.

However, unless I have tests for all this code, it actually drives the coverage down. Sometimes I end up writing extra tests to test Lombok’s code which seems absurd. Other times I replace @Data with the handful of annotations I really need. Some classes need getters on most fields, but only setters on some. After fixing those up, I end up with classes that now have twice as many lines as when I added just the one @Data annotation.

I’ve not really come to a good middle ground on that one. Coverage or clean code. It’s a trade off.

In conclusion

I’ve found SonarQube to be a good spotlight on where my code needs addressing, and have learnt a lot about Java and Java development best practices (according to the “SonarWay”). I was making really good progress on cleaning up some debt and had the bug count down to one! Then I figured out how to fix the missing annotation processing code, and it found 12 new bugs and 91 new code smells. I will endeavour to keep chipping away at it until all the bugs and vulnerabilities are down to zero, and hopefully some debt paid down.

Apart from the coverage dilemma above, SonarQube has been a positive experience. It’s a good way to learn better practices, and be able to chart my progress doing it.

TeamCity WebHooks plugin poll

Posted on Updated on

If you’re a tcWebHooks plugin user, could you please take a couple of minutes to post a comment on this poll?

I am trying to determine how much effort I should invest in supporting older versions of TeamCity, and where to focus future development.

There are some new features in TeamCity 10 which I would like to migrate to, and I am wondering how many tcWebHooks users would be affected by discontinuing support for versions 9.x and below.

Also, how tcWebHooks is being used, and what payload formats are popular.

Many thanks in advance for your time.

tcPrettyEmail updates – branch support and tc9.1.6 fix

Posted on

This last month has seen a couple of updates to the tcPrettyEmail plugin.

One was a bugfix release to support a breaking change in TeamCity 9.1.6 and the other was adding support for feature branch names.

If you’re running your builds from GIT or Mercurial and have your VCS configured for Feature Branches, then tcPrettyEmail will now print the branch name in the subject and in the overview at the beginning of the email.

Downloads are on GitHub and as usual, please let me know if you experience any problems with either of these releases.

New tcWebHooks release for TeamCity 9.1

Posted on Updated on

The recent release of TeamCity 9.1 caused an issue with webhooks failing to be sent under certain specific conditions.
The Branch object supplied by TeamCity as part of an sBuild now contains references to the parent project, which then contain references back to the build which contains a reference to the Branch.

This creates a circular reference which prevented the Xstream library from serialising the webhook payload.

This would only present if one was running a build from a VCS that supported branches (Git or HG) and it was a non-master build (eg, feature branch) and the payload was XML or JSON.

I have worked around this by creating a simple bean and copying just the relevant parts of the branch to the bean which is then serialised in the webhook content.

I also found an issue with creating new webhooks in the _Root project. The “create webhook” message was sent to TeamCity when the webhook editing dialog closed, but the new webhook was never actually being created. This has now been resolved.

Currently, sourceforge is preventing me from adding new release files for download.
I therefore, think now is the time to move to GitHub for releases. The updated release (v0.9.35.72) can be downloaded from https://github.com/tcplugins/tcWebHooks/releases

As usual, any issues can be raised on GitHub or posted on the tcWebHooks bugs page.

Additional Feature: Double template rendering passes.

Messages passed to webhooks payloads containing ${variables} are now parsed twice. This means that custom messages can contain tcWebHook variables as requested in a comment on the blog. You can read more in the commit message.

New tcWebhooks payload format : Tailored JSON

Posted on

Introducing a new payload format called Tailored JSON in tcWebhooks version 0.9.27.61

This format only sends the contents of the “body” parameter as application/json. It is up to the user to define the content of the body parameter.

The easiest way to define the body is with a TeamCity build parameter called webhook.body as per the screenshot below.

Tailored JSON via the body parameter

Building tcWebHooks custom parameters from teamcity build parameters

Posted on Updated on

Custom Parameters are a feature in the tcWebhooks plugin, which allow one to send a custom string to the webhook endpoint made up of a template and variables.

Up until today, those variables have only included other values from the webhook’s payload. However, from now on it is also possible to use build paramters from the TeamCity build configuration. One now has access to the TeamCity build properties. These include properties defined by the build configuration, as well as env.*, teamcity.* and system.* properties.

An example is worth a thousand pictures

      <parameters>
        <param name="aaa_param" value="TeamCity version ${system.teamcity.version} running as user ${env.LOGNAME}" />
      <parameters>

This example parameter will add an item as per the table below.

Name Value
aaa_param TeamCity version 8.1.4 (build 30168) running as user netwolfuk

Using the TeamCity build parameters to create Custom Webhook Parameters

Additionally, any property beginning with the string “webhook.” will automatically be included in the webhook payload with the “webhook.” prefix removed. This allows custom parameters to be declared from the Teamcity build parameters rather than having to edit the webhooks plugin XML file. The above example could then be configured as per the following screenshot.

Screenshot from 2014-09-23 09:27:07
Download the 0.9.26.60 version (or higher) from sourceforge to try it out.

The build parameters configuration in Teamcity is explained on the TeamCity configuration properties wiki page. More info about using tcWebHooks custom parameters is also available.