Mu-Haskell 0.2: Avro and optics
by Alejandro Serrano- •
- February 20, 2020
- •
- haskell• functional programming• functional• mu• microservices• rpc• grpc• protocol buffers• avro
- |
- 6 minutes to read.
Last month, we introduced Mu-Haskell, a library for developing a microservices architecture. Mu takes care of interconnecting the different parts of your application – serialization, transport layer, logging, data sources – so that your team can focus on the business domain.
One of the core design decisions of Mu is the use of service and data schemas as “sources of truth,” which specify the protocol followed by each of the microservice. The 0.1 release included support for Protocol Buffers. In this version 0.2, we add support for Avro.
Mu encourages developers to separate those data types used for serialization and those used in the domain logic, because they often evolve under different constraints. In many cases though, the mapping between both is automated by the library. Alas, we have realized that this approach can be quite heavyweight for simple clients of a service. Since version 0.2, you can also use the smaller API based on optics.
As usual, this new release also comes with many bug fixes, and has led us to contribute in several open source projects for the broader Haskell community. In particular, we have built a parser for Avro IDL from scratch, and we have extended Haskell’s main avro library with support for logical types.
Avro serialization
We intend for Mu-Haskell to be serialization-agnostic; so that servers and clients may be defined in the same way no matter which format is used. However, since version 0.1 only supported Protocol Buffers, this was not obvious. Not any more, since in version 0.2 you can choose your gRPC services to talk using either Protocol Buffers or Avro.
Unfortunately, such support involves one breaking change to our API. Previously, any gRPC client or server was assumed to use Protocol Buffers. Now, a protocol has to be chosen, which results in an additional argument in every function that starts communication:
main = runGRpcApp msgProtoBuf 8080 quickstartServer
The value msgProtoBuf can be replaced with msgAvro above to instruct mu-grpc-server to use Avro. Note that clients and servers written in Mu-Haskell 0.1 are still 99% encoding-compatible with the new version (the remaining 1% stems from correcting a bug in which we deviated from the specification).
Avro has its own language to define schemas and services. This release also makes it possible to use files written in that IDL as sources of information, as supported previously for Protocol Buffers. One single line of Haskell:
avdl "HealthCheckSchema" "HealthCheckService" "schemas" "healthcheck.avdl"
instructs the compiler to load the file healthcheck.avdl from folder schemas and generate type-level schema HealthCheckSchema and service definition HealthCheckService. The reason for using separate arguments for folder and file is that, in Avro IDL, it is quite typical to define common types in a shared file, which is then imported by all those services that use them.
One important difference between Avro and Protocol Buffers is that the latter makes all fields optionals, whereas the former defaults to required fields, and optional ones must be tagged as such. This changes how Haskell data type mapping works a bit, since using Avro doesn’t require wrapping each field in Maybe. Unfortunately, this means that, if you want to expose a service as both Avro and Protocol Buffers, the logic becomes a bit convoluted. We are working on an improved API to handle this case.
Optics-based interface
Having separate data types for serialization and bussiness logic is a good engineering practice. In Mu-Haskell, we try to lower the burden of maintaining these two separate sets of types by (1) taking care of the serialization data types and (2) providing automatic forms of mapping between the types with FromSchema and ToSchema. In some cases though, you just want to explore a service or create a very small server, and, in those cases, the full-fledged approach can be quite heavyweight.
For those scenarios, we have introduced a new API based on the optics package. After adding mu-optics to your dependencies, the fields of your records are exposed as lenses and prisms. You can then use the power of optics to access or modify that information. This is how you create a string for the full name of a person based on its first and last name:
req ^. #firstName <> req ^. #lastName
Those versed in optics may recognize the (^.) operator, which stands for “get.” In short, req ^. #firstName means “access the #firstName field in req.” The use of # is not so common though; we refer to those names prefixed by the aforementioned symbol as labels. By means of the OverloadedLabels extension in GHC, and the Optics.Label module in optics, we can leverage our type-level schemas as a way to know which optics are available for a particular term. In other words, apart from the slightly weird syntax, labels allow us to provide optics for the fields in a type-safe way, but without any Template Haskell involved.
Another place in which optics simplify our code is in the use of gRPC clients. In 0.1, you could either create a record defining all the methods in a service, or use TypeApplications, a GHC extension usually considered advanced. Since version 0.2, methods are exposed as lenses over GRpcClient:
client ^. #newPerson $ record (Just (T.pack nm), Just ag)
In the code above, we are calling the newPerson method, and providing the arguments as a record with two values, namely the name and the age. The function record is also part of the optics-based API, and provides a generic way to build records – in the sense of Protocol Buffers or Avro – without introducing an intermediate Haskell data type.
Conclusion
If you are interested in reading more, head to our docs and examples. With its 0.2 release, Mu-Haskell becomes quite close in feature-parity with its sibling library Mu-Scala. If you want to see another serialization or transport format being supported, don’t be shy and open an issue.
The active development of Mu-Haskell is proudly sponsored by Xebia Functional (formerly 47 Degrees), a Functional Programming consultancy with a focus on the Scala, Kotlin, Swift, and Haskell programming languages.