Introducing Hood and sbt-hood

by Javier De Silóniz Sandino
•
March 19, 2020
•
scala• kotlin• benchmarks• hood• sbt• gradle
|
6 minutes to read.

Announcing the release of Hood and sbt-hood

Hood and sbt-hood are plugins available for Gradle and sbt that allow developers to integrate JMH benchmarks with their CI flow. Many organizations are used to set restrictions in their CI systems so that broken builds or even bad formatted code can block a PR until those issues are fixed. Hood and sbt-hood allow you to do the same based on your own code benchmarks, thus avoiding merging code to your repositories if it brings performance down below an specific threshold.

Basic usage

Hood and sbt-hood behave as any other Gradle/sbt plugin. In order to set it up under sbt, include an entry in your plugins.sbt for it as follows (be sure to use the latest version number for the plugin):

addSbtPlugin("com.47deg" %% "sbt-hood-plugin" % "0.2.6")

If you’re using Gradle, include the following in your build.gradle file (be sure to use the latest version number for the plugin):

buildscript {
    repositories {
        // ...
        maven { url "https://dl.bintray.com/47deg/hood" }
     	//...
    }
    dependencies { classpath 'com.47deg:hood:0.8.0' }
}

apply plugin: "com.47deg.hood"

After that, the plugin exposes a set of settings. The basic ones are as follows:

previousBenchmarkPath: A File instance pointing to a JMH benchmark either in CSV or JSON format. This file will be used as the base set of benchmarks to compare to the current state of your code.
currentBenchmarkPath: Same as before, but pointing to a file containing the most current benchmarks that will be compared to the previous ones.
generalThreshold: With this setting, we can set a global threshold value. This value will be used in all benchmarks comparisons, so if a current benchmark goes below this threshold, it will be considered as a bad result.
benchmarkThreshold: Normally, you’ll want to use multiple thresholds for each set of values in your benchmarks (i.e., maybe you want to treat parsing in a different way than decoding). This setting allows you to set a Map to correspond each different benchmark name with a specific threshold, thus overriding what’s specified in generalThreshold.

These can be configured under the settings value in your build.sbt file, or under the settings for each correspondent task in your build.gradle file.

Finally, the plugin exposes a task to compare your set of benchmarks and output results to the console:

compareBenchmarks

In the following output, you’ll see the different benchmark keys, current and previous filenames and values, and the comparison result based on the configured threshold, like the following:

# ✔ code.parsing (Threshold: 2.0)

|Benchmark|Value|
|---------|-----|
|master.json|476.145|
|current.json|490.096|

Each comparison result will be marked by an emoji. A “checkmark” indicates the comparison was successful and the current value exceeds the previous one. “Caution sign” emoji indicates that the current value is equal or less than the previous one, but it’s still within the configured threshold. A “red circle” emoji will tell you that the current value is below the threshold compared to the previous one.

CI integration

One of the main values you can get from these plugins is the integration you can achieve with your repositories. By enabling some additional settings in your build.sbt or build.gradle files, Hood and sbt-hood can talk to your repositories so that your CI flow can check the current state of your code and block a PR that contains benchmarks that decrease the performance of your applications below what you deem acceptable. These settings are as follows:

token: A GitHub access token. It can be generated by browsing to Developer settings > Personal access tokens within your user settings.
repositoryOwner: User id in GitHub of the owner of the repository where the plugin will post updates.
repositoryName: Name of the repository where the plugin will post updates.
pullRequestNumber: Pull request number where the plugin will post updates.
targetUrl: Optional URL to the CI job, allowing GitHub to show a link to your CI job besides the mergeability status (e.g., TRAVIS_JOB_WEB_URL under TravisCI).

With those set, your CI flow can just execute the following task:

compareBenchmarksCI

This task will execute the compareBenchmarks command (outputting the results to your console log), and then will use GitHub API to submit a comment in the specified PR containing the same comparisons results. It will also flag the PR as unmergeable if any of the comparisons results in an error.

Generating benchmark reports for visualization

Hood and sbt-hood also allow you to save benchmark reports as separate files (to be used in visualization apps like Robeen or JMH Visualizer) and upload them to your repository so you can keep track of your code progress.

Generating report files

You can configure Hood and sbt-hood to generate a report of each benchmark comparison, by just configuring the following settings:

outputToFile: Set to true (it’s disabled by default).
outputPath: Path to the benchmark report to be generated.
outputFormat: File format for the output report (md and json are supported).

Once these are set, both compareBenchmarks and compareBenchmarksCI will save the reports to your desired location.

Uploading benchmark reports to your repositories

If you want to keep track of your benchmark reports, the best way is to ask Hood and sbt-hood to upload them to your repositories. In order to do that, you just need to select the settings already discussed under the CI integration section, together with the following:

benchmarkFiles: A list of report files to be uploaded.
uploadDirectory: Path within your repository where the plugin will upload the report files.
commitMessage: Commit message that will be used for each upload to GitHub.

Then just execute the following task to upload your reports:

uploadBenchmark

Conclusion

Hood and sbt-hood help fill a gap that can be found in many organizations: keeping code benchmarks as a main priority along with code test coverage or code formatting. By customizing these plugins to your project needs, you can detect issues in the performance of your Gradle or SBT based applications and act accordingly before these are merged in your codebase.

Remember to check out both plugins’ websites in order to keep up to date with the latest updates and more thorough documentation.

Hood and sbt-hood are proudly sponsored by 47 Degrees, a Functional Programming consultancy with a focus on the Scala, Kotlin, Swift, and Haskell Programming languages supporting the active development of both plugins.