47 Degrees joins forces with Xebia read more

ValidatedNel in the Wild: Exploring error accumulation in an sbt plugin

ValidatedNel in the Wild: Exploring error accumulation in an sbt plugin

Recently, I fixed a bug in the mu sbt source generation plugin: sbt-mu-srcgen. The issue was that the plugin would silently fail (i.e., exit with a successful status code) when generating valid Scala code if the user provided an Avro schema that used primitive types (e.g., String) instead of record types in any request. Not generating the code for this type of invalid schema wasn’t the problem (since primitive types are forbidden in methods per the mu-scala spec. But the fact that the step appeared to complete successfully, despite not actually succeeding, was a big issue. This meant that, instead of notifying the user of their invalid IDL file at the compile step, the source generation step would complete successfully, and any errors from the missing code would bubble up at runtime. My task was to come up with the best way to fix this behavior and improve the user experience when using the sbt-mu-srcgen plugin. After getting some feedback from the team,

sbt plugin question

I realized that the best approach to fix this behavior would be to accumulate any errors that service found when parsing the Avro schema, and then, based on whether any errors were found, either return them all or exit successfully. In other languages, this type of error accumulation could require me writing a custom validator function and plumbing it throughout the codebase. Fortunately, this plugin is written in Scala, so I could use the Cats library, which has a useful datatype for doing this exact type of behavior: Validated.

A brief interlude on Validated

The reason that Validated is a useful tool for this error handling in Scala stems from the fact that it is an Applicative, which is a typeclass that is frequently used to perform N independent operations. Validation (since it usually consists of validating multiple fields and accumulating errors) is a classic example of an applied Applicative. Furthermore, Validated provides a datatype for error accumulation called ValidatedNel, which is a type alias for Validated[NonEmptyList[E], R] plus some syntactic sugar. ValidatedNel is an extremely useful datatype for error accumulation because it allows us to store errors in a List without throwing away type safety (since ValidatedNel uses Cats’ NonEmptyList under the hood). So, with the context and motivation about these datatypes in mind, I was ready to start using them.

Basic domain modeling

The first thing I did when I started to implement my fix was to do some domain modeling; I wanted to create some type aliases to represent my error-handling data types throughout the codebase. So, I added the following lines to the package object for this service:

// core/src/main/scala/higherkindness/mu/rpc/srcgen/package.scala
+ import cats.data.ValidatedNel

// ...

+ type Error       = String
+ type ErrorsOr[A] = ValidatedNel[Error, A]

Adding those two type aliases would allow me to more reasonably model my error handling throughout the refactor.

Modify the method signature

Type aliases in hand, I went about making the change to the method signature itself. Since I wanted to collect the errors as I mapped over each line of the schema, and either return the NonEmptyList of errors or the List[String] that represented the generated lines of code, I made the following change to the method signature.

// core/src/main/scala/higherkindness/mu/rpc/srcgen/avro/AvroSrcGenerator.scala
  override def generateFrom(
      files: Set[File],
      serializationType: SerializationType
-  ): Seq[(File, String, Seq[String])] =
+  ): List[(File, String, ErrorsOr[List[String]])] =

I prefer this approach (changing the method signature before modifying the internals of the method) because once I’ve altered the method signature, my IDE points out all the areas that I need to fix in my code to comply with the new return type.

Modify the method to match the signature

Once I made the change to my method type signature, I went about modifying the actual logic of the generateFrom method. I started by rewriting an inappropriately-named private function parseMessage (the name was confusing because it was taking structured data and turning it into a string, i.e., doing the opposite of parsing) that had been responsible for reading the schema files and throwing a ParseException on any errors.

To rewrite parseMessage, I broke the schema error handling methods into a request validator and a response validator, and then combined them into one buildMethodSignature method (since ValidatedNel is an applicative, it’s easy to compose). See the below diff for specific details on how I implemented these validators using the validatedNel datatype.

// core/src/main/scala/higherkindness/mu/rpc/srcgen/avro/AvroSrcGenerator.scala
-  private def parseMessage(name: String, request: Schema, response: Schema): String = {
-    val args = request.getFields.asScala
-    if (args.size > 1)
-      throw ParseException("RPC method has more than 1 request parameter")
-    val requestParam = {
-      if (args.isEmpty)
-        s"$DefaultRequestParamName: $EmptyType"
-      else {
-        val arg = args.head
-        if (arg.schema.getType != Schema.Type.RECORD)
-          throw ParseException("RPC method request parameter is not a record type")
-        s"${arg.name}: ${arg.schema.getFullName}"
-      }
-    }
-    val responseParam = {
-      if (response.getType == Schema.Type.NULL) EmptyType
-      else s"${response.getNamespace}.${response.getName}"
-      s"  def $name($requestParam): F[$responseParam]"
-    }

+  private def validateRequest(request: Schema): ErrorsOr[String] = {
+    val requestArgs = request.getFields.asScala
+    requestArgs.toList match {
+      case Nil =>
+        s"$DefaultRequestParamName: $EmptyType".validNel
+      case x :: Nil if x.schema.getType == Schema.Type.RECORD =>
+        s"${requestArgs.head.name}: ${requestArgs.head.schema.getFullName}".validNel
+      case _ :: Nil =>
+        s"RPC method request parameter '${requestArgs.head.name}' has non-record request type '${requestArgs.head.schema.getType}'".invalidNel
+      case _ =>
+        s"RPC method ${request.getName} has more than 1 request parameter".invalidNel
+    }
+  }

+  private def validateResponse(response: Schema): ErrorsOr[String] = {
+    response.getType match {
+      case Schema.Type.NULL =>
+        EmptyType.validNel
+     case Schema.Type.RECORD =>
+        s"${response.getNamespace}.${response.getName}".validNel
+     case _ =>
+        s"RPC method response parameter has non-record response type '${response.getType}'".invalidNel
+    }
+  }

+  def buildMethodSignature(
+      name: String,
+      request: Schema,
+      response: Schema
+  ): ErrorsOr[String] = {
+    (validateRequest(request), validateResponse(response)).mapN {
+      case (requestParam, responseParam) =>
+        s"  def $name($requestParam): F[$responseParam]"
+    }
+  }

Once I converted the parseMessage method to the buildMethodSignature method, I was able to finally build the List[String] that represented the gRPC request without suppressing the ParseException (this suppression had been the root cause of the silent failure, since original implementation involved catching the exception and logging a warning without doing anything with the error). That change looked like this:

// core/src/main/scala/higherkindness/mu/rpc/srcgen/avro/AvroSrcGenerator.scala
- val requestLines = protocol.getMessages.asScala.toSeq.flatMap {
-    case (name, message) =>
-    val comment = Seq(Option(message.getDoc).map(doc => s"  /** $doc */")).flatten
-    try comment ++ Seq(parseMessage(name, message.getRequest, message.getResponse), "")
-    catch {
-        case ParseException(msg) =>
-        logger.warn(s"$msg, cannot be converted to mu: $message")
-        Seq.empty
-    }
- }

+ val requestLines = protocol.getMessages.asScala.toList.flatTraverse {
+      case (name, message) =>
+        val comment = Option(message.getDoc).map(doc => s"  /** $doc */").toList
+        buildMethodSignature(name, message.getRequest, message.getResponse).map { content =>
+          comment ++ List(content, "")
+        }
+ }

After my rewrite, the final result is easier to read AND it accumulates errors, thanks to the power of validatedNel.

Handle the accumulated errors

Finally, now that I’d successfully implemented error validation for a given Avro schema, I also had to introduce error validation for every schema file, since the source generation process mapped over first all of the lines in the schema and then over all of the schema files present in the project. If there were errors in multiple files, I wanted to return each file to the user, along with the error details in each file. This way, library users could immediately know where and why the source generation step had failed.

To do this, I once again reached for the Validated datatype – I decided to model the results of parsing each Avro schema file as a ValidateNel of ValidatedNels. Once I’d modeled the data thusly, writing the logic was easy; one of the great features about ValidatedNel is that you can pattern match against the results, and easily differentiate your logic based on whether you’ve accumulated any errors or not. In addition, since ValidatedNel is an Applicative, it supports useful transformation functors such as traverse and flatTraverse. Below are my changes:

// core/src/main/scala/higherkindness/mu/rpc/srcgen/GeneratorApplication.scala
+ import cats.data.{NonEmptyList, ValidatedNel}
+ import cats.data.Validated.{Invalid, Valid}

// ..

-    generatorsByType(idlType).generateFrom(inputFiles, serializationType).map {
-    case (inputFile, outputFilePath, output) =>
-        val outputFile = new File(outputDir, outputFilePath)
-        logger.info(s"$inputFile -> $outputFile")
-        Option(outputFile.getParentFile).foreach(_.mkdirs())
-        outputFile.write(output)
-        outputFile
+    val result: ValidatedNel[(File, NonEmptyList[Error]), List[(File, List[String])]] =
+    generatorsByType(idlType)
+        .generateFrom(inputFiles, serializationType)
+        .traverse {
+        case (inputFile, outputFilePath, output) =>
+            output match {
+            case Invalid(readErrors) =>
+                (inputFile, readErrors).invalidNel
+            case Valid(content) =>
+                val outputFile = new File(outputDir, outputFilePath)
+                logger.info(s"$inputFile -> $outputFile")
+               (outputFile, content).validNel
+            }
+        }
+    result match {
+    case Invalid(listOfFilesAndReadErrors) =>
+        val formattedErrorMessage = listOfFilesAndReadErrors
+        .map {
+            case (inputFile, errors) =>
+            s"$inputFile has the following errors: ${errors.toList.mkString(", ")}"
+        }
+        .toList
+        .mkString("\n")
+        throw new RuntimeException(
+        s"One or more IDL files are invalid. Error details:\n $formattedErrorMessage"
+        )
+    case Valid(outputFiles) =>
+        outputFiles.map {
+        case (outputFile, content) =>
+            Option(outputFile.getParentFile).foreach(_.mkdirs())
+            outputFile.write(content)
+            outputFile
+        }
    }

As you can see from this example, the final result of my schema reading ended up being a ValidatedNel of ValidatedNels (and other things that didn’t required validation), which made perfect sense considering that I was mapping over every line of every file, and each operation could have N number of errors that could be accumulated into a list.

Wrapping up

Once I re-wrote the Avro schema parsing to accumulate errors via validatedNel instead of relying on ParseExceptions that were getting swallowed, I was able to significantly improve the user experience of the plugin. Before my changes, running sbt muSrcGen against Avro schemas that contained non-record types returned a success message, for example:

[info] Compiling 2 Scala sources to /Users/dylan/Work/playground/test-mu-srcgen-errors/protocol/target/scala-2.13/classes ...
[success] Total time: 7 s, completed Aug 18, 2020, 3:30:41 PM

followed by a runtime error.

Whereas now, any faulty schema will fail at compile time with an error like this:

[error] (protocol / muSrcGen) One or more IDL files are invalid. Error details:
[error]  /Users/dylan/Work/playground/test-mu-srcgen-errors/protocol/target/scala-2.13/resource_managed/main/avro/avro/domain.avdl has the following errors: RPC method request parameter 'user' has non-record request type 'STRING', RPC method response parameter has non-record response type 'STRING'

Now we have useful errors that show up at the compile step! Thanks to the power of validatedNel, I was able to create error handling for malformed Avro schemas that can arbitrarily scale up to any number of faulty files or errors per line. Pretty cool if you ask me.

Big shout-out to my coworkers at 47 Degrees for teaching me about the validatedNel datatype in the first place, and for code-reviewing my initial attempts at a solution. Y’all are awesome. Also, if anyone is curious about what the full PR ended up looking like, you can check that out on Github.

Ensure the success of your project

47 Degrees can work with you to help manage the risks of technology evolution, develop a team of top-tier engaged developers, improve productivity, lower maintenance cost, increase hardware utilization, and improve product quality; all while using the best technologies.