47 Degrees joins forces with Xebia read more

Backend testing taxonomy - Scopes and Techniques

Backend testing taxonomy - Scopes and Techniques

One way to view Testing through a more generalized lens is not only to distinguish by the scope or technique, but also the execution - on how those tests are run - and many others.

In other words, by classifying testing into branches - whether those are common or specific to your business/ use-case - they do not form a flat-hierarchy, but instead build a multi-dimensional view on how functions, subsystems, or entire applications can be tested.

Testing_taxonomy

This blog post goes through a high-level overview of some scopes and techniques in the context of a layered RESTful application with Spring in Kotlin, but these topics are not framework dependent.

Set up

For demonstration purposes, Kotest and Spring’s MockMvc will be used to mock calls to a REST API with the following abstract class TestEnv.

All functions containing a TODO will be explained thoroughly in another blog post. But, for the scope of the content, they can be disregarded.

Using MockMvc avoids building the whole Spring Context, and this allows testing REST controllers without starting the Web Server. Otherwise, this would slow down tests substantially - considering start-up time among others.

TestEnv.kt

import io.kotest.core.spec.style.FunSpec
import org.springframework.http.ResponseEntity
import org.springframework.test.web.servlet.ResultMatcher

abstract class TestEnv : FunSpec() {
  /** remaining code ... **/

  inline fun <reified A> post(url: String, content: String): ResponseEntity<A>? =
    TODO("Mock a POST Call to the REST api using Spring MockMvc")

  inline fun <reified A> get(url: String): ResponseEntity<A>? =
    TODO("Mock a GET Call to the REST api using Spring MockMvc")

  fun postWebsite(post: PostBodyWebsite): ResponseEntity<Website>? =
      post<Website>("/entertainment/website/", TODO("convert post to Json"))

  fun getWebsites(): ResponseEntity<List<Website>>? =
      get<List<Website>>("/entertainment/website/")
}

The following example stores websites with resources concerning various technologies. There are two domain types: Website and PostBodyWebsite. The latter is used to send a POST request to an HTTP API, which in a successful case responds with a Website and is successfully stored.

data class Website(
  val id: Int,
  val reduced: PostBodyWebsite
)

data class PostBodyWebsite(
  val name: String,
  val url: String
)

The Api will have these handles:

  • GET: “/entertainment/website/” returning all stored websites from the persistence layer using a service with an http status 200
  • POST: “/entertainment/website/” saving one website through the persistence layer using a service and returning the saved website with an http status 201

Technique

Regarding testing techniques, there are two common methods - among others: example-based and property-based Testing.

Example-based Testing

Example-based Testing usually follows three steps, which are:

  • Given a specific input of type A
  • Apply the Input to a function f: (A) -> B returning a value of type B
  • Assert that the returned value is of a specific output of type B

This may look like:

import io.kotest.matchers.collections.shouldContain
import io.kotest.matchers.nulls.shouldNotBeNull
import io.kotest.matchers.shouldBe
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.http.HttpStatus
import org.springframework.http.ResponseEntity

@SpringBootTest(properties = ["spring.main.allow-bean-definition-overriding=true"])
@AutoConfigureMockMvc
class WebsiteTests : TestEnv() {
  init {
    test("Get a saved website") {
      val website = PostBodyWebsite("47 Degrees Blog", "https://www.47deg.com/blog/")
      postWebsite(a)
      val response: ResponseEntity<List<Website>> = getWebsites().shouldNotBeNull()
      val retrievedWebsites: List<PostBodyWebsite> = response.body.shouldNotBeNull().map { it.reduced }
      retrievedWebsites.shouldContain(website)
      response.statusCode.shouldBe(HttpStatus.OK)
    }
  }
}

First, a POST Call is run with website, and once all websites are retrieved with a subsequent GET Call, website is expected to be within the list of websites, and the status code of the aforementioned GET request is 200.

The disadvantages of example-based tests are that their exhaustiveness depends on the author, not to mention that edge cases aren’t clearly stated or are covered partially.

Furthermore, those tests are not robust, as they define unclear/ambiguous requirements for a test suite and can’t answer questions like what shape the output or input should have or how the REST API should behave in unexpected scenarios.

Property-based Testing (PBT)

One Kotlin counterpart of QuickCheck, which was one of the first property-based testing libraries, is Kotest implementation kotest-property.

Adding it to our project dependencies, with io.kotest:kotest-property:${Version}, provides us with the combinators we need to define and run property-based tests.

Though each property-based testing library has its differences across languages and ecosystems regarding features and primitives, they allow users to write testing protocols, which are run against Generators users can customize.

Let’s have a look at an example:

First, define a Generator for a PostBodyWebsite.

import io.kotest.property.arbitrary.bind
import java.net.URL
import io.kotest.property.arbitrary.string
import io.kotest.property.Arb
import io.kotest.property.arbitrary.filter
import io.kotest.property.arbitrary.map

/** URL Generator **/
fun Arb.Companion.url(): Arb<URL> =
  bind(
    of("https", "ftp", "file"),
    string().filter { "[-a-zA-Z0-9+&@#/%?=~_|!,.;]*".toRegex().matches(it) },
    string().filter { "[-a-zA-Z0-9+&@#/%=~_|]".toRegex().matches(it) }
  ) { scheme, authority, path ->
    URL("$scheme://$authority/$path")
  }

/** Here we document what a favorable shape of this type should look like in code, e.g.: the generated website names can't be empty **/
fun Arb.Companion.website(): Arb<PostBodyWebsite> =
  bind(string(minSize = 1), url().map { it.toString() }, ::PostBodyWebsite)

Second, define a property that is:

A previously saved website, which is randomly generated with Arb.website(), is included in a subsequent GET request, and the status code for all GET requests for websites is 200.

import io.kotest.matchers.nulls.shouldBeInstanceOf
import io.kotest.property.Arb
import io.kotest.property.forAll
import org.springframework.http.HttpStatus

test("Website GET Request results with a `200` and includes a previously saved website") {
  forAll(Arb.website()) { a: PostBodyWebsite ->
    postWebsite(a)
    val response: ResponseEntity<List<Website>> = getWebsites().shouldBeInstanceOf()
    val retrievedWebsites: List<PostBodyWebsite> = response.body.shouldBeInstanceOf<List<Website>>().map { it.reduced }
    response.statusCode == HttpStatus.OK && retrievedWebsites.contains(a)
  }
}

In addition to quantifiers like forAll and forNone, there are ways to encode exhaustiveness and shrinking modes to find a ‘smaller’ reproducible sample of an error, but this will be covered in a dedicated post around PBT.

In contrast to example-based testing, property-based testing does not only come with the support of running tests against a plethora of Input Data, but shines when asserting stateful properties, such as the latter, or stateless properties as in the following - testing a costume URL Generator Arb.url():

import io.kotest.core.spec.style.StringSpec
import io.kotest.property.Arb
import io.kotest.property.arbitrary.filter
import io.kotest.property.arbitrary.of
import io.kotest.property.arbitrary.string
import io.kotest.property.checkAll
import java.net.URL

class UrlSpec : StringSpec({
  "Url doesn't cause MalformedURLException" {
    checkAll(
      Arb.of("https", "ftp", "file"),
      Arb.string().filter { "[-a-zA-Z0-9+&@#/%?=~_|!,.;]*".toRegex().matches(it) },
      Arb.string().filter { "[-a-zA-Z0-9+&@#/%=~_|]".toRegex().matches(it) }
    ) { scheme, authority, path ->
      URL("$scheme://$authority/$path")
    }
  }
})

Despite covering a significant class of use-cases in testing, property-based testing comes with a few shortcomings.

In example-based and property-based tests, authors define the state and how subsequent operations or events follow to a favorable or unfavorable outcome.

In other words, they are always in control of the program state.

When our goal is to abstract over a multitude of states and their identities, there are other techniques we won’t cover here in this blog post, but are worth mentioning.

These involve, for example, verifying API operations with an explicit state model checker. These tools answer questions, whether a set of operations in a REST API can dead-lock for a specific use-case and output a chain of events to how such a situation appears, or other interesting questions surrounding concurrent environments, such as liveness properties - concerning assertions that extend infinitely into the future, and many others.

Scope

Let’s shift the focus now from techniques to scopes.

Unit testing

Up until now, each layer of this application is not tested in isolation, without depending on the whole infrastructure and other unrelated dependencies.

Let’s test the website controller, which depends on a service WebsiteAlgebra, that accesses the persistence layer, the Database here, to perform Crud operations.

import org.springframework.web.bind.annotation.RequestMapping
import org.springframework.web.bind.annotation.RestController
import org.springframework.web.bind.annotation.PostMapping
import org.springframework.web.bind.annotation.GetMapping
import java.util.concurrent.CompletableFuture

/**
 * Spring's Bean mechanism can autowire the implementation of [WebsiteAlgebra], where we access the Database, into the constructor below.
 */
@RestController
@RequestMapping("/entertainment/")
class WebsiteController(
  val websiteAlgebra: WebsiteAlgebra
) {
  @PostMapping("/website")
  fun saveWebsite(
    @RequestBody website: PostBodyWebsite,
    request: HttpServletRequest
  ): CompletableFuture<ResponseEntity<*>> =
    TODO()

  @GetMapping("/website")
  fun findAllWebsites(): CompletableFuture<ResponseEntity<List<Website>>> =
    TODO()
}

One way to isolate this dependency in Unit tests is to define a @TestConfiguration, where we define a dummy representation of WebsiteAlgebra.

WebsiteUnitTestConfig.k

import org.springframework.boot.test.context.TestConfiguration
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Primary

@TestConfiguration
class WebsiteUnitTestConfig {

  /**
  * Annotating this dependency with [Primary] is important, to ensure that there aren't any ambiguities.
  */
  @Bean
  @Primary
  fun dummyWebsiteAlgebra(): WebsiteAlgebra =
    TODO()
}

Then import WebsiteUnitTestConfig into the Spring Context.

import org.springframework.context.annotation.Import
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc
import org.springframework.boot.test.context.SpringBootTest

@Import(WebsiteUnitTestConfig::class)
@SpringBootTest(properties = ["spring.main.allow-bean-definition-overriding=true"])
@AutoConfigureMockMvc
class WebsiteUnitTests : TestEnv() {
  /** define our property-based tests ... **/
}

This is one way to enforce the inversion of control, where a dummy representation/ test double can be implemented in different variations, which is explored more thoroughly in a dedicated post on Unit testing.

The view of a single unit may differ, depending on the project and defined scope. The following example is a minimal test that generated PostBodyWebsite, which is successfully saved for arbitrary sample sets in Arb.website().

import io.kotest.matchers.nulls.shouldNotBeNull
import io.kotest.property.Arb
import io.kotest.property.forAll
import org.springframework.http.HttpStatus

test("A generated website can be saved successfully with a status 201") {
  forAll(Arb.website()) { a: PostBodyWebsite ->
    val response: ResponseEntity<Website?> = postWebsite(a).shouldNotBeNull()
    val website: PostBodyWebsite = response.body.shouldNotBeNull().reduced
    response.statusCode == HttpStatus.CREATED && website == a
  }
}

Integration testing

In contrast, integration tests focus on the very opposite of what we separated above. They test the interaction of different modules, layers, and are more concerned about the flow of data.

With integration testing, assertions are centered on/around integration points, rather than the units that were scoped in the previous paragraph.

Let’s add another endpoint to the Rest API that fetches websites based on their id or returns a null if no element can be found with that id.

This is a mocked call to that endpoint:

TestEnv.kt

fun getWebsiteByIdOrNull(id: Int): ResponseEntity<Website?>? =
    get<Website?>("/entertainment/website/$id")

Here’s a property-based test:

test("A saved website from the generator `Arb.website()` can be retrieved by id") {
  forAll(Arb.website()) { a: PostBodyWebsite ->
    val id: Int = postWebsite(a)?.body?.id.shouldNotBeNull()
    val response =
      getWebsiteByIdOrNull(id)
        .shouldBeInstanceOf<ResponseEntity<Website>>()
    val result = response.body.shouldBeInstanceOf<Website>()
    result.reduced == a && result.id == id
  }
}

Depending on the scope of integration tests, sometimes it is too costly to spin up all modules and services for a test suite. Here we can utilize the inversion of control to speed up tests.

Pact testing

Furthermore, there is a need for consumer-driven contracts, as several architectures become more and more distributed, service-oriented, and communicate with other APIs.

Pact tests verify producers in the application against mocked producers, which other teams/consumers can test against, thereby assuring an aforementioned aspect in integration tests, which allow mocking service producers or consumers.

It not only validates assumptions about an application from a consumer perspective, but can verify API migrations and updates as the application evolves.

In addition, this technique enables CI to assert control over producers that impact consumers.

This topic will be addressed in upcoming dedicated posts, introducing a bunch of terminology surrounding pact testing and an in-depth example.

Infrastructure testing: Chaos testing

This sub-series focuses on validating and building more confidence around properties in an uncertain distributed environment.

The goal is to learn how an application recovers from chaos experiments using ArrowFx coroutines, functional error handling, and circuit-breakers, among other methods and data types.

We will use tools and libraries that concern chaos engineering, injecting step-by-step controlled failures/experiments into endpoints, and establish them in CI as test suits.

In a way, we are preparing an application and ourselves against wildfires in production.

Sum up

In this blog post, we briefly touched on the following taxonomies:

  • Technique:
    • Example-based testing ✅
    • Property-based testing
  • Scope:
    • Unit testing
    • Integration testing
    • Pact testing
    • One form of infrastructure testing that is chaos testing

Stay tuned! This post just teases some topics in an ongoing series about Backend testing in Kotlin.

Our next post in this series is focused on property-based testing.

Ensure the success of your project

47 Degrees can work with you to help manage the risks of technology evolution, develop a team of top-tier engaged developers, improve productivity, lower maintenance cost, increase hardware utilization, and improve product quality; all while using the best technologies.