ETags and Concurrency with RESTful Microservices

Today I was talking to a colleague and puzzling over the nuances of concurrency with our microservices. We got to the bottom of it, so I wanted to take a moment to jot down what we concluded before I sleep and forget it all.

Imagine a fictitious car insurance quoting domain. There are three microservices for this example and each has its own database. Each has its own REST API that I’ll refer to as “the front door”. Each is connected to an event bus and they announce when their data has changed.

Imagine that each a design tenet is to have the microservices use a real REST API (as opposed to one of those ones that doesn’t use hyperlinks to self-describe) that takes the client on a journey through getting some business objective done, like quick-quoting car insurance.

Because of the hyperlinking, the journey would take place by interacting entirely with a single service endpoint, and by endpoint I mean HTTP server host, or base URL. In other words, I should not have to do any of the work against a different host.

Each microservice is therefore standalone and its possible to complete a journey if other microservices are down. To make this possible, some microservices may keep a non-authoritative copy of salient data.

For example, a car insurance quote may need to keep the quote number, vehicle information and driver details, and preparing a quote might mean verifying the vehicle and its type, while there are dedicated microservices for dealing with master list of all known vehicles and types, and all known drivers and their driving histories.

Microservices are free to collect and cache information that they depend on to go about their jobs, but they are not considered the authority on that foreign data.

Being a real REST API, the resources drive the journey by offering up hyperlinks, and some of these contain a JSON approximation of an HTML Form with the crucial addition of an ifMatch key-value pair within the hyperlink structure. Sort of like this. Well, exactly like this.

{
   "title": "Begin Quote",
   "description": "Starts a new quoting worklow.",
   "href": "/api/values",
   "rel": "start-quote",
   "method": "POST",
   "ifMatch": "l56nvhk32c6j5hv7l65n867m",
   "form":
   [
       { "id": "quoteId", "name": "quoteId", "options": [ "127390179" ] },
       { "id": "title", "name": "fullName", "options": [ "Mr", "Mrs", "Etc" ] },
       { "id": "forenames", "name": "fullName" },
       { "id": "surname", "name": "surname" }
   ]
}

Preparing a quick quote may involve a few steps, each a form-filling exercise, performed solely against the Insured Vehicle Service API. The workflow may involve a driver search and select perhaps on license ID. The drivers are foreign data, cached in the Insured Vehicles Service from driver update events originating from the Insured Drivers Service.

Preparing a quote means that the Insured Vehicle Service ends up with some information about the vehicle, some about the quote itself like the premium and the driver(s).

The creation of the quote is announced to the world via an event and the Insured Drivers Service observes and creates a new driver while the Insurance Quotes Service creates a new quote and maybe an activity log.

What’s interesting is how the version entropy, or ETag, is managed for the data we hold. In a document database, if all the information for a quote is stored within the same doc, we can recompute new entropy with each change.

What’s important is that when the IVS observes a change to non-authoritative data it holds, then it updates its copy and the entropy for the quote changes, too.

Should a form be submitted via the front door of the IVS, and between the form being rendered in the resource and the form received, a concurrent change is made via the front door API of the IDS, then the IVS updates its cached copy on the quote. Now the ETag for the quote will no longer match and the form will be rejected, the consumer will need to refetch the resource.

If we imagine the IVS API has a form for adding an additional driver on a quote, but there’s a rule that young additional drivers is disallowed, then the above behavior is desirable; a correction of a driver date of birth would invalidate the ETag of the doc storing the data in the IVS and the form would be rejected with Precondition Failed. In some situations when the client refetches, a link might disappear completely since the action is no longer a valid move.

The logic involved in adding an “add additional driver” form link to a resource and the default data in the form is based on information we hold about the drivers in the household. Thus, the form derives from the quote and driver data but perhaps not the vehicle data since we may not have a rule about the type of car and additional drivers.

With this in mind, if we used SQL, or we stored these cached foreign things in different documents then each one would need its own version entropy.

Importantly, when rendering the form we’d need to compute the ETag, i.e. the ifMatch that goes alongside the form, from the entropy in all the source documents or rows from which we derive information, perhaps by XOR-ing it.

Before I finish, I’d recommend not breaking a domain like this until its absolutely necessary and consider building strictly isolated (both code, rules and data) but in-process services that communicate in a loosely-coupled way via “normal” synchronous coded events, callbacks or delegates and move them out one-by-one to their own processes and introduce the asynch event bus slowly.