Semantics Misunderstandings Cause Integration Risk
The majority of integration risks lie in understanding semantics. This risk can easily be mitigated through simple communication patterns.
Consider a common scenario where one team in an organization has to consume a service from another part of the organization. Between the producer and consumer lie two main things:
The architectural layers facilitating the service
The shared understanding of how the service is intended to be consumed
Navigating the architectural layers can be treated as a fixed cost as an organization will have a a handful of patterns though which services are exposed and consumed. It’s a learn once, apply many times over pattern. For example, once you figure out how the API gateway works and learn how to pass in credentials, the same pattern applies in other integrations. Even if you had to do 10 more such integrations these patterns remain relatively static. Perhaps you might switch from using JWTs to OAuth 2.0 when using another service, but there’s only a small number of such patterns to deal with.
The shared understanding of how a service is consumed is where the majority of the risk lies. This category of risk can be further broken down:
A shared understanding of the shape of the data elements used in communicating
The semantic understanding of those data elements
The toolset to describe the shape of the data has always been a first-order concern. XSD schemas described SOAP services, Swagger describes JSON APIs, and GraphQL offers powerful means to probe what elements are available and how exactly to retrieve them. The toolset has taken the mystery out of what data types to expect when invoking services.
It is the second of this category of risk, the semantic understanding, that does not have a technical solution, and thus is where the majority of integration difficulties lie. For example, a Swagger can tell you to expect a string value for payment_code
when calling a service, but it won’t tell you what exactly the various values mean from a business perspective. Good, up-to-date documentation can help reduce this risk and Stripe’s documentation is Exhibit A in how one might go about approaching this. Creating good documentation is a skill and takes time to create and maintain. However, in the absence of the rare documentation that is self-evident and bullet proof, there is are approaches to reducing the risk resulting from semantic misunderstandings of the APIs within an organization: talk to each other.
As simple as a solution that may seem, the following classic anti-pattern appears to be prevalent:
Instead of the people (i.e., developers) that create the services and the people (i.e., more developers) that consumer the services, we have installed layers of unnecessary communication protocols that dilute communication in favour of structure. This is an idea rooted in (sometimes) good intentions. The idea goes something like this: to optimize developer time and prevent them from getting distracted, organizations install mechanisms to funnel communication between teams, with the conduit’s (usually multiple managers) ability, time, and understanding of the context being the bottleneck in communication between the provider and consumer.
Conservatively speaking, this bottleneck increases the cost and risk of integrations by 50x. What should have been a simple conversation between two teams that know what they’re doing, now involves 5-6 other people who have little idea of the nitty gritty details of what’s actually happening.
People working with these teams should recognize the pattern of people who are not part of either team trying to chase information for the team. As coaches, our goal here is to create structures where the developers on the two teams are able to directly and safely communicate. Any other pattern should be questioned until it is proven to mitigate a risk that the two teams’ developers couldn’t have mitigated otherwise.