Introducing Mu: A purely functional library for building microservices
by Juan Pedro Moreno Estudillo- •
- January 22, 2019
- •
- mu• scala• rpc• http• avro• protobuff• protocol• buffers• json• openapi• microservices
- |
- 10 minutes to read.
We are very excited to announce that we’ve renamed and expanded the frees-rpc library into Mu, version 0.17.0.
Freestyle RPC was born as a purely functional library for building RPC endpoint-based services, atop gRPC and Freestyle. Now, in this new stage, Mu has opened its scope, becoming a functional framework for building software architectures based on microservices.
Why Mu?
"Mu": it is the Greek letter following "Lambda", and the symbol for "micro". There is some overlap in the usage of this symbol across areas like mathematics, engineering, and physics. For Mu, it represents a purely-functional microservices-based architecture taking into account communications, serialization, reliability, protocols, and schema evolution compatibility.
Higherkindness
Mu is now part of Higherkindness, an ecosystem of user-focused libraries for FP including:
- skeuomorph: library for transforming different schemas in Scala. It provides schema definitions as non-recursive ADTs, and transformations & optimizations via recursion schemes, based on droste.
- rules_scala: robust and featureful Bazel rules for Scala.
- compendium: HTTP service for schema storage and client generation.
There’s much more to come in Higherkindness, and we’ll be sharing more on these existing libraries in the near future, so stay tuned.
Mu overview
As you know, some benefits of Functional Programming include code that’s easier to reason with, test, debug, and maintain since everything you write can be utilized in future software pieces, preserving the referential transparency, and allowing us to focus on inputs and outputs. This can be perfectly extended to your system architecture, where the communication side effects need to happen, and matter.
Mu brings the capacity to model all your service edges based on Tagless Final algebras, where we don’t need to worry about the underlying protocol or serialization format initially. Basically, every service in your world receives an input or a stream of inputs and returns an output or stream of outputs, nothing else. That’s the main principle of Mu, and in practice it supports:
- Communication layer: RPC endpoint-based, and HTTP coming very soon.
- Serialization format: JSON, Avro, and Protocol Buffers.
- Protocol support and code generation: Avro, Protocol Buffers, and Open API (coming soon).
- Integrations for metrics reporting (Prometheus and Dropwizard).
- And much more.
All these capabilities make Mu a complete and ideal framework for microservice environments, no matter how they are exposed (RPC, REST).
Mu Science
Mu is protocol/specification oriented; the business rules should remain the same regardless of implementation or of how the bytes travel through the wire. In other words, the only thing that changes when using different serialization formats or protocol types is the specification language or IDL that is utilized, so no significant changes in the code are needed.
Protocols can be seen as analogous to functions. In terms of referential transparency, a protocol is no different from a function accepting an input value and returning an output value. It should return the same output given the same input without affecting code elsewhere in the system.
* Picture obtained from the amazing blog post The Science Behind Functional Programming by Rafa Paradela
That would be the essence of the services and messages protocols in your system if they are described using a specification language or IDL. The IDLs describe an interface or protocol in a language-independent way that Mu is able to understand (Avro, Protocol Buffers and Open API), enabling communication between software components that do not share one language.
Let’s look at three tiny examples, where the Scala representation is the same in every case:
- Interface defined with Protocol Buffers:
service Greeter {
rpc sayHello (HelloRequest) returns (HelloResponse);
}
message HelloRequest {
string name = 1;
}
message HelloResponse {
string message = 1;
}
- Interface defined with Avro:
@namespace("higherkindness.mu")
protocol Greeter {
record HelloRequest {
string name;
}
error HelloResponse {
string message;
}
HelloResponse sayHello(HelloRequest greeting);
}
- Interface defined with Open API:
/greeter/hello/{name}:
get:
tags:
- metadata
description:
Say hello
operationId: greeter
parameters:
- name: dataset
in: path
description: 'Name of the person.'
required: true
example: "John"
schema:
type: string
responses:
'200':
description:
Hello message.
content:
application/json:
schema:
type: string
What would Scala representation look like for this kind of protocol? Mu will generate something like this:
case class HelloRequest(name: String)
case class HelloResponse(message: String)
@service(// either [Protobuf], [Avro] or [Http])
trait Greeter[F[_]] {
def sayHello(request: HelloRequest): F[HelloResponse]
}
At a high level, with this definition on the server side, you could just add an implementation for the Greeter service, bind it to the server, and spin it up. That’s it.
On the client side, you would just need to import the IDL definition, compile the project, and use the auto-derived client for the Greeter service.
Mu Stack
Apart from main libraries: gRPC and Http4s, Mu is entirely based on FP-oriented libraries like cats, cats-effect, fs2, fs2Grpc, monix, monocle, pureconfig, circe, as well as others that are very important in the serialization format side, like avrohugger, avro4s, and pbdirect, among others. Mu wouldn’t be possible without the great Scala community.
Mu Modules
The multitude of features and capabilities introduces a big drawback, the transitive dependencies. However, the project aims to introduce as few dependencies as possible in your build, so the risk of binary incompatibilities is lower. In order to accomplish this, Mu provides different sbt modules that you can use depending on the case. Although you can find all the specs in the readme file of the project, let’s list some of the main modules:
For RPC-based systems (server-side):
mu-rpc-server: needed to attach RPC Services and spin up an RPC Server.mu-rpc-prometheus-server: optional. Scala interceptors which can be used to monitor gRPC services using Prometheus, on the server side.
For RPC-based systems (client-side):
mu-rpc-channel: needed to auto-derive clients to be able to talk to the server.mu-rpc-nettyormu-rpc-okhttp: one of both, depending on the transport layer.mu-rpc-prometheus-client: optional. Scala interceptors which can be used to monitor gRPC services using Prometheus, on the client side.
For RPC-based systems, you can optionally add some other common artifacts on both sides (server and client):
mu-config: optional. Provides configuration helpers for loading the application configuration values.mu-rpc-netty-ssl: optional. Needed when securing communications with SSL, where only Netty is supported.
Mu brings the same streaming capabilities that gRPC does: server streaming, client streaming, and bidirectional streaming. Furthermore, it enriches the type system offered in favor of purity, type safety, and referential transparency. For example, the gRPC Java types for streaming are converted into monix.reactive.Observable or fs2.Stream types, which you can choose depending on your needs. To do so, you would import either mu-rpc-monix or mu-rpc-fs2 in your build.
Mu Environment Setup
Previously, we described some of the components and artifacts that make up Mu, as well as the different serialization formats that are currently supported. When generating servers or clients, everything is configurable and customizable thanks to the sbt plugin: sbt-mu-idlgen. This plugin provides settings such as:
IDL Type, where the available values are currentlyAvro,Protobuf,OpenAPI.Serialization format, coveringAvro,Protocol BuffersandJSON.Source: where the IDL definitions are placed.Target: the target path in your build where the Scala files will be located after the plugin generates the sources.- And much more.
Mu in Action - Use Cases
Like most engineering problems, there isn’t a single solution for challenges related to communication systems. Mu seeks to unify the way you set up these communications across the network: use REST when it makes sense, or use RPC if it is more appropriate, or use both and have the best of both worlds! Everything is under the same umbrella: Mu.
In the following diagrams, we are trying to represent a couple of use cases based on Mu; the first one is quite simple, the other one, a bit more complex.
Mu Future
In addition to upcoming features such as mu-http and code generation from Protocol Buffers files, Mu is in the process of being ported to Kotlin, using Arrow as a powerful FP implementation.
Summary
Check out the Mu documentation to learn more about the project and see some additional examples, and visit the Mu Gitter channel for more information, questions, and to jump in on the conversation. We also encourage you to stay tuned for updates on our blog, where we will be presenting some of the more common use cases very soon.
As always, we’re looking for new contributors! All levels are welcome and we encourage you to reach out with any questions on involvement. You can read Mu’s Contributing Guidelines to get started.