Overall evaluation

Could read

High level thoughts

This is a book that didn’t match what I thought it would be coming into it. I was geared up to see a series of patterns of API design, each with a series of statements about how the pattern impacts key quality considerations such as understandability and evolvability. I thought I would be able to see a variety of different way to solve potential problems that had different trade-offs attached. What I think I found instead was more of a taxonomy of parts of an API design, and references other works that may have fit my expectations better.

Patterns

I still have a copy of Design Patterns, Elements of Reusable Object-Oriented Software on my bookshelf. This is a book that is criticised today as potentially leading new developers down paths that are unhelpful, and even at the time was criticised as containing patterns that were more about working around limitations in popular programming languages of the day than about thinking in objects.

I have a different view. I think as a young developer it really did help me get a handle on the kinds of ways to use objects. Few probably remember now, but we were coming from a world of heavy procedural design and were often thinking about data in terms of abstract data types and creating the early equivalent of interfaces by way of structs containing function pointers. Having languages that treated type as well as class, or interfaces a first-order concept was novel and there was a lot of confusion about what “good” looked like. I think a number of the patterns in that book don’t work in practice, but I think the book helped line the industry up on more or less the right kind of track.

API Design

Service and product APIs have now also been taken up as first-order concepts both in commercially-available and internal products. It makes so much sense these days to design API-first or at least to take an API-centric approach. For machine to machine integration purposes the API is the product, and even when the “real” product is a user interface or the like - always having a well-defined and well-governed API helps us manage conceptual boundaries between systems as well as trust boundaries and other separation of concerns.

When I say “API first” I’m really saying things like “API before business logic”, and “API before database”. Within a DDD frame of mind what we really want to do is effectively model the user’s thought process about how our services work and what they do within their own business domain so that we can consistently offer a low-friction, surprise-free experience. We want our APIs to surprise our customers only in ways that bring about a feeling of delight: “oh, I didn’t realise it would be this easy…”.

That’s a difficult thing to do, and that is why we get paid the big bucks really. We want to be able to provide that end to end experience to our customers of

  • learning about the existence of our API,
  • understanding what it can do for them,
  • supporting their integration in a low-friction way,
  • providing effective feedback and support during and after integration,
  • not giving them bill shock
  • operating the service with high reliability, low latency, and high throughput
  • managing consistency in a way that doesn’t surprise
  • matching privacy and security expectations
  • not breaking their integration

Is this an API-centric book?

It feels to me like this is not a book built with an API-centric approach in mind. For example, instead of using OpenAPI throughout the book we see a few limited snippets of OpenAPI and largely instead see MDSL. When we abstract away from the thing we are trying to make into a first-class element of our systems I think we have automatically demoted it to second class. I think MDSL could be a useful syntax for academic articles where it is important to get a lot of information across in a small space. If that is the intention here I don’t disagree with its use of stereotyped templates. However, if we are hoping to present a practical approach to API design then I think we see that not fully working out even within this title.

We see another attempt to abstract the API from itself in the Context Representation pattern which encourages us to build our APIs in a way that promotes “protocol independence”. And while that can be useful in some circumstances I think tends to take our eye off the ball in a sense as we try to design the best API possible for our users who are accessing services at across a specific protocol. What I would have perhaps preferred to see here is some kind of description of data dictionaries, each with a clear description of how they might relate to specific protocols. However, even any such data description is only ever going to be fully correct within a specific bounded context.

One of the quotes that really stood out to me while reading from an API-centric perspective was in the “How it works” section of the Data Element pattern.

Define a dedicated vocabulary of Data Elements for request and response messages that wraps and/or maps the relevant parts of the data in the business logic of an API implementation.

I’m sure I’m reading too much into it but really stood out to me as a description that feels application-centric rather than API-centric. What I think the author might have been going for here or should have been is more along the lines of…

Define a domain representing the business or service as seen through the eyes of your customer. Map into and out of that domain as required to address internal design considerations and to support the API as an antifouling layer between an external customer and any corresponding internal representation.

Are the right patterns included?

It feels to me that this book presents more of a taxonomy of parts of an API than it presents patterns for addressing or trading off various design concerns. I think the patterns get stronger the further we read into the book, but I want to point out patterns that if feel are defined in a tautological way. That is to say, that the pattern is more or less a restatement of an underlying business requirement.

Frontend and Backend Integration

Frontend integration is a pattern where an API connects to a front-end application. That I think essentially boils down to the concept of governing the message exchanges between a frontend and backend, which is a really important concept if you have any kind of security concerns or other non-functional concerns to address. So in that sense I think this is a useful concept. However, taking a step back it feels a little weak to me. You have a frontend and a backend, and you want to connect them. That’s the pattern. A pattern usually refers to a design decision we are able to make, but this one feels like a statement of requirements on the API before any design decisions have been taken.

We then contrast this with backend integration, which is a statement of the requirement “I have two backends and I want to connect them”.

What we could have talked about instead are things like when we should have a dedicated API and backend for our frontend app, versus when it should integrate directly against a backend service layer. This is hinted at, but I feel like it should have been the central design decision we talk about for the frontend integration case. Likewise, for direct integration - is it better to have one big API or many small ones? What are the impacts on API evolvability versus comprehensibility? Could we find a balance where specific user personas or use cases are addressed by individual small APIs, or does that make things more complicate for users who don’t neatly fit a single persona or have many use cases?

Public, Community, and Solution Internal APIs

I think the distinction between these API types again feels more like a requirements question than an API design question. “Who am I trying to reach and serve with this API?”. Community API feels a bit weird to me. Perhaps this is because I would normally think of private connectivity for specific customers as an add-on offering to an API that does not really change its design in a meaningful way.

What we could have talked about instead is a series of patterns for customer onboarding and customer identification within the API design space.

Basic message structure

These patterns of “Atomic parameter”, “Atomic parameter list”, “parameter tree”, and “parameter forest” feel to me a bit like we are saying “json exists”, or even “structured data exists”. In one sense it feels a little too basic, but in another it feels under-described.

What I wanted to read about in these patterns were the impacts of message structure decisions on API evolvability and comprehensibility. For example, if we have a list of strings, and we find that we want to add fields to it then we end up with two problems:

  • We aren’t able to add new fields within each list entry because we chose an atomic rather than a tree or dictionary type
  • We are going to have trouble referring reliably to entries in the list because they will tend to move around

I think we also don’t really know from any of the description in these patterns whether our list is likely to be ordered and if so - how?

I think there actually relatively few cases in API design where list structures or plain atomic-parameter documents stand the test of time as APIs evolve.

Define Endpoint Types and Operations

Just as the Basic Message Structure patterns feel like “just JSON” to me, the endpoint types and operations feel a little like “just CRUD”. I think there is enough depth in this section to be worth talking through, but it isn’t like we can actually trade off between these types most of the time. I think most of the time our requirements are going to define which one we use. Perhaps the most interesting part of this section is a discussion that I think was buried a little about whether a REST-style resource which operations based on state transfer operations is ideal for every use case versus when a non-state-transfer operation makes more sense.

API Key

One of the patterns I like least in the book is API Key. It is presented as an authentication pattern, but I would tend to say that a build-your-own API key solution is very much a security antipattern. Moreover, it is not situated in the book alongside other potential authentication patterns.

If we want to talk about the API key concept independent of an authentication discussion then we could talk about it instead as an identifier for a given integration. There is a great pattern hiding here in the discussion where a customer who has a given set of actions they are able to perform instead on their own behalf creates an API token that identifies the client accurately but has a restricted set of actions they are able to perform. This makes least-privilege something the user can self-service and would have been good to cover as a top-level pattern here.

If we do want to talk about security, then I think we ought also to be talking about privacy of message contents and which additional parties might have access to a given message that the user doesn’t know about.

Error report

I love this pattern, but I didn’t love the examples. For starters, I think we need to distinguish between an error report we return because of a system error versus a report we deliver because of an integration error.

The following distinct error reports should be called out:

  • Integration error
    • The customer integration is not valid
    • We want to return very helpful information about what the customer needs to do to repair their integration
    • We do not want the customer system to couple itself to the specifics of these error messages as they are likely to change over time.
  • System or upstream error
    • The system failed in some way, or failed to communicate with an external party
    • Depending on the cause, we may want to count our SLA impact differently
    • Customer systems should be tested against this failure type to ensure they deal with it correctly
  • Refusal
    • We want to be careful to distinguish between failures and refusals
    • We don’t want to count refusal to perform an operation negatively in our SLA
    • We do want customer systems to be able to deal with refusal
    • Refusal is likely to follow a distinct response structure

One thing that really stood out to me in the examples was the missing correlation identifier. We see timestamps in the examples, but we don’t see a clear system identifier for the failed request that was included in each log message associated with that request. That means when the customer calls us up they won’t be able to communicate to us exactly which request failed, and we’ll have to go diving in the logs in a way that is really unnecessarily difficult for everyone involved.

Refine Message Design for Quality

I think some patterns in this section are good and not in need of much further elaboration. I would have liked to see a bit of discussion of flow control and the ability to control load on the system as part of “Request Bundle”. Conditional request is straight from HTTP and should cover both optimistic locking requests and cache validity checking. The conversation around Embedded Entity versus Linked Information Holder is among the most “pattern-like” patterns in the book where the reader is encouraged to consider design tradeoffs between these two.

I think Wish List and Wish Template are not necessarily distinct patterns, and what maybe I would have liked to see instead is a discussion of the degree to which we should allow users to specify arbitrary queries. It’s easy to expose security and other vulnerabilities by exposing a full query language, but defining our own limited language which when then need to extend multiple times carries with it its own risks and issues.

I think Metadata Element is a bit light in its treatment of important types of metadata, and in particular is missing any description of obligations the recipient of information might have under applicable law.

Evolve APIs

In a sense this was one of the more comprehensive sections where various approaches to API versioning are considered. What I didn’t see enough of, and which I think is our core obligation as API owners who are keeping evolvability in mind, is the impact of patterns of backwards-compatibility and forward-compatibility without versioning. Tolerant Reader is mentioned in this section, but not actually called out as a pattern, which I found surprising.

I think most of us operate in an ecosystem where we can’t constantly break customer integrations. Demanding that customers keep pace with our release schedule means that they have to keep directing a portion of their development capacity to our API, even if they aren’t adopting new features. I think we need to take care to retain customers who have existing integrations and relatively simple needs.

To me this largely means identifying the lifetime of any given API and restricting the jargon we allow into that API based on it expected lifetime. If it only has to last a few months we can pretty much put in whatever we like. If the API has to last a few years then we have to start putting firm limits on content and concepts we include in the API that are particularly application-specific. If it has to last ten years then the API had better follow data schemas and semantics that have been stable at an industry level for at least a good fraction of that lifetime. It may be useful to segment the user-based into multiple APIs depending on how long each API needs to be able to live for.

Again I feel like APIs take a second-class role in this section. What I am describing here is where we use the requirements we have on how long an API needs to last in order to design that API appropriately. Whereas what I find in the book feels much more service-centric where the needs of the user are not necessarily central to the narrative.

Document API Contracts

I find this another weird section for me. More than any other section of the book so far the patterns “Service Level Agreement”, “Pricing Plan”, and “Rate Limit” speak to the relationship we have with a given customer and how the design of the service we offer them may impact the experience. So then, where is the “Support call” pattern? Where is the pattern of logging the API capabilities in use so that we can discover which we can turn off and which are bringing home the bacon?

“Rate limit” talks about how we might want to restrict the usage patterns of a given authenticated customer, but doesn’t really address questions like what happens when a customer has a bug that introduces an accidental authenticated denial of service attack or how we might prioritise traffic or manage flow control or auto-scale when the system is under stress.

I would have liked to see a stronger focus on how to deal with and describe consistency questions between cloud regions or on-prem datacenters at an API level. Overall I think this section was a fair surface level description of some of the things we’d like our customers to know before they start using an API.

Are the APIs in this book expertly designed?

I think as a book on API design, the times APIs are actually described we’re going to want to put them through a bit of extra scrutiny.

The first such description appears in chapter 2 as the Target API Specification. This is written in MDSL so is a bit hard to review in terms of API quality concerns. I like that CustomerID is of type ID, but it isn’t totally clear at this level of abstraction what and ID is. I don’t like seeing “CustomerResponseDto”, which makes it feel to me like the API here was built after a java implementation was already in place. I’d like to see AddressDto changed to refer to some kind of international standard definition if possible, and it is clearly missing “country”. The “phoneNumber” in CustomerProfileUpdateRequestDto seems likely inadequate, and it is unclear if an international standard is being followed or not. The use of “firstName” and “lastName” likely doesn’t address the needs of international customers. Versioning the CustomerInformationHolder endpoint is a bit curious. Are we versioning the media type in use or the operation, or the whole API?

I think the next time we see an API is in chapter 3 where we learn that a birthday is a timestamp rather than an ISO date, so our customer was born at 1989-12-31T23:00:00.000+0000 . The reality is that a birthdate needs to be a date. If you treat it as anything else you are going to end up with interoperability problems when you communicate that timestamp to another system that convert it to a date in a different way. The error report in this section is missing a correlation ID as I have mentioned above so will make support calls more difficult. A later message has creationDate as a timestamp again, so is it a date or a timestamp? How will the software interpret it I wonder, and will it survive conversion to another format and back again? The problem applies to startDate and endDate for the policy period.

In Chapter 4 I like the self link but dislike the fact that the authoritative permanent link for a given entity would have an extraneous query parameter. Is this the “self” link of the customer that was returned? It seems not to be, even though it is returned as part of that structure. If this is a query for a single customer, why is there a list included? I’m puzzled by the “address.change” link in the Atomic Parameter section. Is this a resource that represents the address of the customer, and would support GET and PUT or is it a form submission endpoint that would only accept a POST? Under “Parameter Tree” I see a numeric customerId field, which must be a no-no in all but the simplest services these days. As part of “moveHistory” we see both from and to addresses, which will presumably introduce either duplication or introduce confusion. It is likely better to model this the other way around, where a given address has a from and to date. The “when” in this structure of “2022/01/01” doesn’t follow an international standard.

In chapter 6 we see for the first time I think an OpenAPI specification that was generated from MDSL, which seems to be the cause of the description containing fields “anonymous1” through “anonymous6”. I can forgive field names like “requestContextSharedByAllOperations” in a descriptive textbooks setting, but I’m less forgiving of a URI path “/CustomerInformationHolderService”. I’m not sure how CustomerInformationHolderService is a tag useful for distinguishing resources in the description either, and would have though that “customer” and “information-holder-service” would at least be distinct tags. Arrays of strings should be avoided for evolvability reasons. Session ID as an integer seems as short-sighted as any other integer identifier and should be avoided.

Message granularity in Chapter 7 follows a different structure for links as compared to previous chapters. Link structures should be as standard as possible if we are expecting a machine to be able to navigate hyperlinks.

String validation is missing throughout all the examples, leaving these APIs unnecessarily vulnerable to XSS attacks.

Conclusion

This is a book that talks through a number of fundamental elements of API design mostly as taxonomy but at times introduces patterns that reflect legitimate design decisions and tradeoffs. I feel this is more of an academic work than one I want to hand to my developers, but I think it will be worth a read for some of you.

Disclosures

  • I have written for Pearson in the past, see SOA with REST
  • One of my SOA with REST co-authors is an author on this title
  • The copy I read was provided to me free of charge

Configuration Identification