Sound advice - blog

REST is a state of the art technology for service-orientation and related disciplines. It and its related architectures define a set of constraints that support the principles of

• Standardised Service Contract,
• Service Loose Coupling,
• Service Abstraction,
• Service Reusability,
• Service Autonomy,
• Service Statelessness,
• Service Discoverability, and
• Service Composability.

Runtime discoverability a core property of a REST architecture, and is achieved through its Uniform Interface and use of hyperlinks (URLs in documents). Governance is applied separately at two distinct levels in the architecture:

1. At an architecture-wide level. Document types, communication patterns, and resultant standard methods are standardised. This is known as the Uniform Interface. A standard document type on the Web is HTML. A standard communication pattern would be to issue requests to get a document until one is successful. A standard resultant method would be the "get" method.
2. At a service or service inventory level, standardising the service contract in terms of a set of fine-grained identifiers (typically URLs) with individual capabilities and meaning of their fine-grained capabilities. Fine-grained capabilities conform to the architecture-wide standards. An example fine-grained identifier would be for the current state of a specific purchase order or employee record. As URLs these might be "https://orders.example.com/purchase/123" and "https://employees.example.com/bob".

Coupling of services and their clients is primarily to the architecture-wide standard document types and methods, and is minimised between services and their clients. Every client and resource are expected to implement every appropriate pattern of communication and every appropriate document type for transferring information. Thus, any client and service that it makes sense to plug together can be plugged together at runtime by specifying the appropriate fine-grained identifier and invoking methods on it. Any client that expects to download a purchase order and perform reporting on it can be pointed at runtime to a designated purchase order URL and work correctly.

REST communication patterns and their resultant standard methods are:

• Client-Server, as are classical SOA services
• Stateless, with a state deferred by transferring it back to the client after each request.

The communication patterns support:

• Caching, with implicit or explicit labelling of responses as cacheable or non-cacheable. If a response is cacheable, then a client cache is given the right to reuse that response data for later, equivalent requests.
• Layering, with the full fine-grained identifier passed into any request. A proxy acts as a shared server for one or more clients, taking requests and forwarding them, with possible translation, to services.

Document types in REST typically include hyperlinks: Embedded URLs that allow the client to interact with another resource capability in either the same service or another service, discovering these new resources at run-time. Embedded scripts or other program data may be included for execution within an appropriately-secured sandbox.

By applying principles from the Web, REST architectures can evolve gradually over time. The following mechanisms support a rapid pace of architectural change, however never require the simultaneous upgrade of services or their clients:

• URLs are decoupled from methods and document types, and evolve separately. Redirection is supported in communication patterns to allow logical identifiers to change over time as required.
• Document types are decoupled from URLs and from methods, and evolve separately.
  • Must-ignore semantics on documents support the addition of information in future versions of a document without the need for versioning or explicit switches. This allows a legacy service to accept "future" versions of the document based on the controlled evolution of a single document type.
  • Content negotiation is supported in communication patterns to support must-understand content in new document versions. These new versions are identified differently to the old version, and parallel generation and parsing implementations must be supported in order to ensure interoperability of legacy and new services within the architecture.
• Methods are decoupled from URLs and document types, and evolve separately. A means for rejecting methods that are not understood or ignoring extensions to understood methods is supported in communication patterns. These features allow fall-back to existing behaviour where new communication patterns are not supported by both parties to a conversation. However, this axis of communication is more difficult to change than the other two. Therefore, a small set of general patterns is preferred over a large set of more domain-specific or architecture-specific methods.

The high level of interoperability and runtime-discoverability intrinsic to a REST architecture supports a number of additional features. Caching is built into communication patterns, allowing this feature to improve the efficiency of the architecture through caching proxies and other devices. Load balancing, high availability, and high scalability are inherently straightforward to achieve compared to a classical SOA.

An individual REST architecture has a defined set of methods and document types. However, architectures can overlap. A Web architecture can be extended with new document types and methods to form a more domain-specific architecture. Existing Web tools are able to operate in this domain-specific environment as far as commonality exists between the two. Due to the decoupling between methods and document types, many components that are agnostic to document type operate correctly in the domain-specific environment without any modification.

Benjamin

I have been reading up a bit on Service-Orientation, lately. Notably, I have been reading SOA, Principles of Service Design by Thomas Erl. I have not finished this book as yet, but have found it quite interesting an enlightening. I think I could reuse about 80% of it to produce an excellent primer for REST architecture ;) The objectives in my view are very similar.

Object-oriented and functional decomposition

One aspect that struck me, however, was the suggestion of layering services upon each other to effectively abstract common functionality. This book in particular describes a task, an entity, and utility layer. These layers describe a "uses" relationship, in that one service or layer cannot operate without the next layer down being available. A task layer service that implements a particular business process will not be able to operate unless the entity services on which it depends are available. The task and entity services will not be able to operate unless all of the utility services on which they depend are available. Service authors are encouraged to reuse capabilities present in lower-layered services in order to avoid duplication of effort.

This is somewhat of a classical Object-Oriented approach, and one that is mirrored in the object-oriented view of systems engineering (OOSEM): You talk to your stakeholders and analyse their requirements to derive an ontology. This ontology defines classes and the capabilities of these classes, alongside tasks and other useful entities. These become the components of your system.

However classical this may be in the software world, I understand the more classical model of systems engineering to be "function"-based. This functional model is again derived from conversations with your stakeholders and your analysis of their requirements. However, it follows a common (inputs, outputs, processing) structure for each function. The main logical diagram for this kind of decomposition is a data-flow diagram. Functions are then allocated to system components as is seen fit by the technology expert, and interfaces elaborated from the flows of data.

Perhaps it is true to say that the OOSEM approach focuses on the static information within the system and its capabilities, while the functional model focuses on the pathways of information around the logical system. In this way, perhaps they are complimentary styles. However, my instinctive preference is for the primary logical system model to based on data flow.

I think that the REST style is amenable to this kind of system design. The emphasis of a REST architecture is also on the transfer of information (ie state) around a system. Functions in the logical description of the system should map cleanly to REST services that consume, and in their turn produce data for onward consumption.

An ontological decomposition of the system should also be taking place, of course. These mappings of domain entity relationships will shape the set of URLs for a given service, the set of domain-specific document types, and onto the internal models of individual services.

Reuse of services and products

I think there is more life in the function-based or modular approach than software engineers might give it credit for. It explicitly doesn't encourage dependencies within the architecture. Architects are encouraged to allocate functions cleanly to components, encouraging cohesive functionality to be co-located and more easily decoupled functionality to be allocated to different services or components. I think it is reasonable to look at this as a highest-level view of the architecture, or a least a useful view at the same level as other architecture. A term that might click with SOA architects is "application". I think that it is important to be clear what applications your architecture is offering for its customers, even if these are not directly embodied in specific services.

I think it is also worth talking about a separation between product and solution spaces when it comes to discussing reuse. We obviously do want to reuse as much code and as many components as we can. However we do not want to do this at the expense of the future evolution of the architecture. Let's assume that part of the system is owned by organisation A, and another by organisation B. The solution maps to the services operated by the two organisations, while products support these services. Different services may be based on the same technology or different, and may have matching or mismatched versions. If there are dependencies between these two groups we need to be careful that:

• There an adequate process for requesting enhancements. Will the release timelines of entity and utility services match the requirements of higher level services? How are competing requirements and release timeframes from different user organisations going to be dealt with?
• That the ongoing availability of the used services is able to be guaranteed, not only at a technical level but also at an organisational level. Who is going to pay for it, and will the inevitable political games that follow eventually bring about its downfall and the downfall of the architecture?

I think there is an argument for avoiding these kind of inter-organisation dependencies, and to control inter-organisation interfaces with reasonable care. Code or component reuse between organisations can be expressed as products commonly used by both, rather than requiring a uses relationship to exist between organisations in the solution.

A product picked up and used as a service within multiple organisations will still need a development path that deals with customer enhancement requests. However, each organisation will at least be able to manage the rate at which it picks up new releases. The organisation will also be able to manage the availability aspects of the service independently of the needs of other organisations using the same product.

I guess this is the fundamental source of my discomfort with the kind of reuse being described. When some people talk about REST scaling, the mean caching and performance of a given service. I think more in terms of multiple "agencies", ie individuals or organisations who own different parts of a large architecture and deploy their components at times of their choosing based on their own individual requirements. A REST architecture is centrally governed in terms of available communication patterns and document types, but does not require central control over the services or components themselves or of their detailed interfaces and URL-spaces. This can be delegated to individual organisations or sets of organisations within the domain.

Conclusion

At the technical level at any given point in time, an optimum architecture will contain little to no duplication of functionality. However, we have to consider how the architecture will evolve over time and also consider the social and organisational context of an architecture. Reuse may not always match the needs of an architecture over time, and reuse of services would seem to exacerbate the barriers associated with achieving reuse. Individual application and service owners should at least be in a position to control their set of dependencies, and be permitted to develop or use alternatives when a supplier organisation is not keeping up with their needs.

Considerations of reuse should be balanced with the need to minimise dependencies and control interfaces between organisations. Reuse of underlying products should be considered as an alternative to direct reuse of services across current or potential future organisational boundaries.

Benjamin

My last article introduced the application of model-view-controller to REST client and server design. I thought I would expand on that material a little further, and provide some diagrams as food for thought.

The REST Triangle

The first thing I would like to do is re-introduce the REST triangle, roughly in line with the terminology I have been preferring of late.

When I first introduced the triangle we were all talking about nouns and verbs, and the obviousness of preferring to have more nouns than verbs. Honestly, I think it caused as much confusion as it did clarity. The obviousness is hard to draw from a set of natural languages that have a lot of both nouns and verbs, and the analogy doesn't really help people come to grips with the impact on their architecture.

Instead I have been tending to talk about Resources and Patterns, URLs and Protocols. Resources are network-exposed objects, and tend not to be that easy to grasp until you relate them directly to the concept of URLs. People get URLs. They understand what they look like and roughly what they are for. If you can stick to that end of the story you shouldn't get too much blow-back.

I prefer Patterns to "verbs". I think the verb or method concept is too low level, and doesn't really give an architectural feel for why limiting them is useful. Patterns on the other hand give in my view about the right feel for the real architectural motivation. People understand that you want to catalogue and reuse patterns. Every message exchange is part of some sort of pattern when you operate over an unreliable network. There are always message retries or redirection, or an imperative not to retry, etc. Communication at this level is hard for the uninitiated, and keeping to a fixed set of well-understood set of patterns only makes sense.

Part of the Uniform Interface constraint is that patterns are baked into protocols. You don't talk about a GET pattern, then make up a new method every time you apply it. You make the method name match the pattern. This bypasses much of the service discovery jiggerypokery that you would otherwise have to do. If you already know what pattern you want to apply, and the server supports that pattern, then you know which method to call. After that it is just a matter of making sure you have the right URL in hand to complete the request.

It almost goes without saying, but I'll say it anyway: Any pattern in REST architecture must of course conform to REST principles. That does not mean that the set of patterns is fixed by any means, but we have gotten away with surprisingly few patterns so far.

Squeezed away in the corner there you might call Content Types, but that doesn't really mean much to many people. I tend to call them document types these days. That has a reasonable mapping for most people to what we really mean when we talk about document types: An Excel document isn't much use to Word. A Word document isn't much used to Powerpoint. They are different document types.

The separation of Document Types and the patterns that transfer documents around the network is evolvability gold. You can introduce new kinds of information into the architecture as new document types without breaking your existing infrastructure for moving documents around. However, we also want to catalogue and reuse Document Types. We want to reuse our parsers and generators. We want to reuse our human understanding of what a particular document actually means. Your patterns will likely even let you negotiate between client and server to ensure that everyone can relate the conversation to a document type they were designed to understand.

Coding the Triangle

By no means is there only one way to implement a service or a client. One of the big selling points of REST and SOA generally is that they can operate in a heterogeneous environment where different technology and approaches are used on either side of the client/server divide. With that in mind, let's look at what I would consider a basic "body plan" for a REST client and server. I'll focus on an application of the GET pattern for purposes of example, but the same plan fits any pattern you would care to mention.

Model-view-controller as a pattern means many things to many people in many different contexts. Its exact meaning has been somewhat eroded over time and replaced with specific variants for special situations. The overall goal of the pattern, however, has not changed: Isolate appearance from implementation. Separate application logic from the constraints of how that logic is presented to users and to other applications.

Down the middle of my plan I have placed the corners of my REST Triangle. I have the set of Document Types in the architecture. I have the set of Patterns and Protocols. I have the set of Resources or URLs in the architecture. These are abstract entities included to give the diagram a bit of context. Each resource participates in a defined subset of the architecture's overall set of Patterns and protocols, and provides the public face to application logic behind the scenes between client and server.

Down the bottom right and left corners of the diagram I have the client and server model components. These components rightly do not depend on anything and do not play any public role in the message exchange. They are shielded from view by their respective Controllers.

The client controller includes code to pilot a pattern's state machine from its initial to its terminal state, despite possible adverse network conditions along the way. Lost response? Retry the request. Redirection? OK, follow if it looks reasonable. Fail the pattern if it looks dodgy. Positive response from the server? You beauty. Process the response.

The role of the server controller is quite different. It will include objects in its scaffolding to handle requests to any of its Resources. That might mean one object per resource. More likely it will mean an object tree with parameters along the path that collect a number of different resources together. These parameters will be passed into the appropriate function on a scaffolding object.

Let's say our request was a GET to https://example.com/light-bulb/1234. Chances are the request would arrive at a function internally identified as something like Resources.light_bulb.get(id). The controller component will then consult its model as part of fulfilling the request:

Resources.light_bulb.get(accept, id):
	bulb = Model.get_bulb(id);
	return Transform.to_acceptable(accept, bulb.status);

As well as consulting the model, this server consults its set of transforms. It wants to return information about the light bulb in a form that the client understands. Therefore, it uses the accept header from its request and passes it through to the Server Transforms component. Multiple controller functions may return the same Document Type and make use of the same transformations to prepare the returned document.

Once returned, the client will also apply its own transform to the document. Its objective is to map the document into data structures that be easily applied to its own model.

Conclusion

Client and Server Controllers deal with the set of Resources and the application of Patterns and Protocols. The Models of each will be queried and updated as part of the transaction, and the Server Model contributes to the set of Resources by containing objects that can be looked up through the Server Controller. Transformations at both end relate documents to model-ready data, and model data to documents.

One of the big questions I think the REST software community are facing at the moment relates to what REST support we should expect from vendors, and what good REST support looks like. I have been taking a lead from web frameworks such as Ruby on Rails, and Gears.

Model-View-Controller means many things to many people. Most of those people will agree that it is a good idea, even if they aren't quite sure what it means for them in their environment.

A classical GUI application would employ application logic in a model, and its view would be a set of widgets on the screen. A controller object either intercedes when events come in from one or the other, or links the two together at display call-up time.

The prevailing wisdom in the world of server-side Web frameworks is more something along the lines of:

• The model is a set of objects that live within a database
• The view is a set of templates for documents that will be returned as part of requests
• The controller is a mapping out of the URL space, which calls out to the model and view components as required

This works well in the common environs of web servers today. Anything that is strictly application logic that you might want to port to another framework one day appears in the model. Any routing of requests, filtering, or other request handling operations appear in the controller. Views allow complex web pages to be built in a template form that the controller can populate with model data.

Success on the Server side: The Controller

I have to say that I am a fan of request routing of the form found in Rails. It is appealing to me to map a GET of https://example.com/objects/{id} easily and efficiently to the getObject(id) function within the controller. It gives a reasonable degree of flexibility in relating URLs to internal controller objects, and places the list of URLs exposed by the application and their methods in an single easily-accessible and readable place. The rails implementation might include a little bit too much magic under the covers, but the central idea to my mind is clearly valuable.

Success on the Server side: The View

The views component is also useful in talking about how we move from RESTful web sites to RESTful services. The biggest change I would make here is to look at it as a two way set of transformations. It is just as important to easily process the document from a PUT request as it is to be able to produce a document in response to a GET request. Machine-to-machine communications common to services also tend to be more structured than HTML. The concept of a template per se is of diminishing use as we move headlong into structured XML payloads, so it is probably more important to include a generic data mapping tool something along the lines of an Altova Mapforce.

Success on the Server side: The Model

The model is perhaps the most interesting component of the MVC pattern. In many ways the model could be the whole of a classical SOA service. Previously you would have had (say) an EJB Session Bean as the front-end of your service. You would expose methods on this bean that operate as a thin controller object over the entities in your model.

It seems like writing a well-designed REST service involves writing an additional layer on top of the classical SOA service. You add a controller and its set of data mappings to what you might otherwise have exposed directly to the network. In doing so you gain the benefits of a uniform interface. The implementation is still Object-Oriented, even if the interface is uniform. Just like the implementation of an object's method is still rooted in the structured programming world while the interface to the object is clearly O-O. It is the vendor products that ease the development of this additional layer of abstraction that will be most useful for REST service development.

Success on the Client Side

The client-side interfaces that provide the most leverage will be the ones that cleanly separate data mapping from communication patterns, and most cleanly expose the patterns themselves to users. Built into these patterns will be any retry mechanisms, redirection, content negotiation, etc. The client won't have to deal with individual HTTP response codes. These will be baked in.

Consider a standard GET pattern. The client application code will simply say: I want data from that URL over there, and I'm using this data mapping to interpret the results. The data mapping will inform the pattern which document types it supports as input, and the pattern will do the rest. It will either reach a success or fail terminal state, and inform the application code client of that fact.

Benjamin

Central to the REST architectural style is the concept of a uniform interface. The typical elements of the interface are a common set of methods and document types used throughout the architecture to interact with a set of resources.

Lately, I have been preferring to focus on communication patterns rather than methods as elements of the uniform interface. REST patterns are different to Object-Oriented patterns in that they typically do not permit the methods that implement the pattern to change from usage to usage. An Object-Oriented observer pattern would use an implementation-defined subscribe method to register a client's interest, and an implementation-defined call-back method to indicate changes to the client. A publish/subscribe mechanism in REST would use a standard SUBSCRIBE method, and a standard notification method back to the client.

In this way, the constrained set of methods in REST are emergent from the communication patterns in use within the architecture. You have a GET method? Well, you probably have a pattern that makes use of it. Also included in the pattern is all of the valid response codes, reasonable expectations about how redirection and other response codes may be handled, and implicit or explicit mechanisms to recover from a failure.

I see a pattern involving GET requests that allows a client to obtain data from the server in a form the client understands. I see a pattern involving PUT and DELETE requests that allows a client to transfer its changing intent for a piece of server-owned state. I see a pattern for creating a set of server-side resources. I see a pattern involving publish/subscribe for keeping a client up to date with server-side data. I even see borderline REST patterns for reliably queueing or tunnelling sequences of messages to the server.

In REST patterns it is the URLs that change, providing contextual meaning for the interaction. Applying the GET pattern to a particular URL only has meaning if we know that we are checking the state of a lightbulb in order to provide early warning of failure. The PUT pattern only makes sense when we know we are turning the lightbulb on or off. In classical object-orientation this information might be held in the type of the server object. The role of resources in REST moves the emphasis of a pattern from an abstract choreography to a concept more like the use of defined base classes or interface classes.

REST patterns are typically implemented in terms of methods, response codes, and headers. They are agnostic to content and the specific URLs they operate on. They transfer opaque documents according to their mechanics to nominated URL end-points. The set of patterns in a given architecture should be small and well-defined in terms of its intent and its mechanics.

It is generally the client that drives a pattern to its ultimate conclusion. Server code tends to be passive, processing requests as they come in. Understanding the patterns that are present in your architecture will help you produce effective and relevant APIs for client to drive the pattern mechanics. The patterns should help you determine which mechanics should live within the API, and which the application code should be responsible for.

Document your patterns, today!

Benjamin

I committed a few more bytes to the network, and have document what I think is the very core set of REST patterns. These cover from a patterns perspective many of the REST constraints. Only layering is called out explicitly as a pattern:

1. GET
2. PUT
3. Registry
4. Layering

Do have a read. Each of the patterns tries to cover the whole lifecycle of a series of requests and discuss the pros and cons of the choices made. I include information about what is important to supply in requests and responses, and how clients and servers should generally behave. Hopefully I'll be telling someone something they don't know. I'm trying to create something of a bridge between the practical and the theoretical in the REST space.

I haven't documented code on demand, the so-called "optional" constraint (which makes it sound like a pattern to me) of REST. I'll start working on others in the coming weeks and may cover it then. I welcome suggestions, contributions, critique, etc.

Benjamin

I have been starting to mentally collect a few common REST patterns of late. I have hacked up a little alternate feed where I plan to place some of these patterns.

The very phrase "pattern" might be a little strong for some of what I have in mind. Often we are talking about very simple structures of client and server, or more of an organisational construct to support a REST architecture. However, I will be attempting to document important trade-offs in any given pattern.

The first pattern I have attempted to document is GET. I do not cover all possible permuations of GET (I see some of these permutations a separate patterns). Instead, I have focused on the usage where a client wants to retrieve a defined small-ish set of data from its server.

Once all of the possible failure cases and the possibility of differently-aged software components in the architecture are considered, GET is actually fairly subtle set of optimisations. I have attempted to show the fundamental communication that is going on between client and server, then map it onto specific HTTP protocol elements.

I have been trying to follow the style of the Gang of Four's Design Patterns. However, I would generally expect that most of the patterns I will describe will be implemented in protocol libraries rather than requiring actual application-level code to participate in significant ways. Comments are welcome.

Benjamin

So I have a document at a given URL, say "https://example.com/doc". It's a big document, and clients need a consistent synchronised replica of this document. The problem is that this document changes frequently, but only in small ways. It make sense that a client will have to grab a copy of the whole document initially, but do they really need to grab the whole document over and over as it changes? What options exist? Is "Delta Encoding in HTTP" the right answer, or is something else needed?

Straight HTTP

Let's start with the HTTP/1.1 of rfc2616. This protocol lets us go and fetch the data initially. Along with that fetch we might get an ETag, a Last-Modified, and/or explicit cache control mechanisms. These let us avoid transferring the document again if it has not changed. If it has changed, however, we must acquire the whole thing again.

Delta Encoding in HTTP

rfc3229 introduces "Delta encoding". This allows us to make a request that says: "Hey, I have the representation with this ETag already. Can you give me the diff against my document to catch up to the current representation?". It looks a little like this:

GET https://example.com/doc HTTP/1.1
If-None-Match: "123xyz"
A-IM: diffe

This says that the client is willing to accept a patch in "diff -e" format from the server. The client can apply this patch against the document it already has to update it to the current revision. A special 226 IM Used response indicates that the sever has understood the request and knows what to do with it.

The general idea would be that the server keeps either copies or diffs relating to several recent versions of a representation. At request time the server will combine recent diffs into an update that brings the client up to date from its current ETag. If it happens to ask for an ETag that is too old, it will simply get the whole document back again.

The model is still nicely stateless. Any number of clients might sitting on the same Etag-identified document version. The server doesn't try to track individual clients, so should be able to handle a serious number of them. The server's only important decision is how long to make the queue. This could relate to a fixed amount of available storage space, a fixed number of document revisions, a fixed time range, an unbounded revision control history with manual purging, or even perhaps some dynamic behaviour based on requests made to date. Interestingly, I suppose, only revisions that were actually transmitted to someone need to appear in the list of diffs.

The delta encoding RFC is a proposed standard, but seems to have a number of problems in this usage profile:

• It's a bit complicated.
• It seems to have to work pretty hard to deal with existing caches that don't understand the protocol extension.
• It seems to have a fair bit of overlap with features of rfc2616, introducing unnecessary alternative ways to do the same thing. It seems so closely coupled with rfc2616 that it should not really be a separate specification,
• It invents a seemingly redundant register for patch format names. Notably, it doesn't appear to allow mime types as patch formats.
• It doesn't seem to be widely implemented.

So what's the alternative?

Multi-URL approach

The two basic approaches here are a one-URL solution or an "n"-URL solution. rfc 3229 uses a one-URL approach, where different diffs are returned to different clients from the main resource URL. An "n"-URL solution would give a unique URL to each diff. Adding URLs seems to be the answer to most problems in the REST domain, so let's have a go at applying it to this one.

The first request

The first request is always going to return the whole document. Instead of simply returning an ETag, consider the possibility that a response will contain a hyperlink. This hyperlink would be to the document that will become the diff against this revision of the resource to update it to the "current" revision at that time. An example response:

HTTP/1.1 200 OK
Link: <https://example.com/doc/deltas?last=123xyz>; rel="Deltas"
Content-Type: ...
...

This rel-deltas link might tell an aware client that it can acquire deltas against the current version by querying the listed URL. The client may use this URL for the second request, below. An Etag and other cache control information may also be included for non-delta-aware clients.

The second request

When the client suspects that the resource has changed it will issue a GET request to the Deltas URL. It will specify the patch formats it understands as mime types in an Accept header, and otherwise the request will be as normal:

GET https://example.com/doc/deltas?last=123xyz HTTP/1.1
Accept: ...
...

The response that comes back might depend on whether changes have occured, and on whether the delta can be generated. The normal delta case:

HTTP/1.1 200 OK
Link: <https://example.com/doc/deltas?last=456abc>; rel="Next"
Content-Type: ...
...

This would deliver the delta to the client in a format the client understands. Normal content negotiation and other features of GET are employed and various kinds of errors can be reported. Some errors would leave the client in the dark as to how it should proceed, and the client would fall back to fetching from the main URL again.

The "Next" link is important. We can't have a client that has successfully applied this delta to ask for its delta from the same place again. Other clients may need this URL, and the client needs to move on to another. In this case the client should move onto the "next" delta in the series for its next request.

Here we see the case where nothing has changed, yet:

HTTP/1.1 204 No Content

Nice and easy. Nothing has changed, so nothing to see here. Cache controls are likely to be the same as for the main URL, but might differ. The client should try again next time it suspects something might have changed.

Finally, we have the case where the delta we are trying to fetch is too old:

HTTP/1.1 410 Gone

Gone is appropriate here, as this URL is no longer expected to work for anyone. However, clients must also be ready to treat a generic 404 Not Found in the same way. The client would be expected to fall back to querying the main URL, restarting the state synchronisation from a current state.

Subsequent Requests

Subsequent requests would hop from next to next as required by the client, and as the client anticipates a change may have occurred.

There is actually a fairly deep question in this approach as to whether the client should follow a Delta or Next link immediately or not. The answer is potentially tied up in caching.

We theoretically want caches to work well with this approach. Now, caches might be smart enough to apply patches themselves and feed them downstream. On the other hand, we might be dealing with caches that are deployed today. Rather than treating the caches as smart relays, we could potentially treat them as "dumb" independent relays for main URL and delta URL responses.

The fact that we want the caches to work for us means that we don't want to set no-cache directives on delta URLs. The data retrieved by any given client may therefore be out of date when it arrives, within freshness criteria attached to the delta. This criteria will either cause the cache to refetch, revalidate, or return data without revalidating. This information is combined with client preferences to determine whether the client is actually returned the freshest possible data or not.

However we do this, I think we can trust the existing caching models for dealing with deltas. Whenever we are given a delta URL the assumption should be that until proven otherwise there is no data there. If the caching model on the data that supplied the new delta URL says that data can be a few seconds out of date, the link will also be a few seconds out of date. If the model says we must revalidate, we'll get something more up to date.

Conclusion

Some advantages to the alternative approach include:

• The server handling main URL requests and the server handling deltas could trivially be separated. The "Delta" and "Next" URLs are not constrained as to where they point.
• The set of delta formats supported is not limited to be predefined. Any mime type (by local or global convention) will work.
• Various GET headers are trivially unambiguous in their operation, and headers that we might not anticipate being useful on the delta (including future protocol extensions) can be used as appropriate nonetheless.
• A Delta might itself have a Delta link. The fact that we could implement recursive deltas a demonstration of additional theoretical power to this approach. Is that power useful? Who knows?

I am fast forming the view that a few links in HTTP headers will be more effective than an implementation of 3229. I think this approach avoids complication at proxies, avoids introduction of new return codes and headers, and will be more generally applicable. For example, it could be used to cheaply transmit new atom feed entries to clients who aren't quite up to date rather than just keep a large document in sync.

Benjamin

This is the final chapter in a series I have been running on my evolving understanding of 4+1 view architectural descriptions. This time around I am covering scenarios, and other things that might end up in an architectural description. We have already established the set of components in our architecture, and the functional interfaces to external systems. We have drawn links between components, but not really elaborated on where responsibilities lie between a given pairing of linked components.

Scenarios

The scenarios view seems to be the most flexible of the views, and seems to capture "other". I think I am sticking fairly closely to the Architectural Blueprints definition when I use this view primarily to convey behaviours of components and internal interfaces. I have been using collaboration diagrams showing component interactions in arrangements that roughly correlate to the Process View presentation. As with other views I have generally tried to align things to the Logical View classification, and attempted to come up with one or more scenarios for each logical function.

Recording of sales is a fairly straightforward transfer of Sales Records from Web Browser to the Sales Manager Component. Financial Transactions pertaining to these records are submitted to the General Ledger for further analysis.

It is possible that we could talk more about the user interface than I have in this diagram. We could talk about how the browser navigates to the page that lets them enter the Sales Records. I have deliberately kept things fairly simple here, trying to focus on clarifying roles within the architecture rather than getting too tied up in the detail of various interfaces involved. Even my method invocations don't look right from a UML perspective. They are really more along the lines of data flows, and I don't expect a literal analogue to exist for any of these method invocations.

Another way to approach the Scenarios View would be to take things back up a notch. Use cases may better capture the end user's intent, whether they be UML-style use cases or more general Systems Engineering -style use cases. I think there is a lot of flexibility in this area, which makes things difficult to nail down. I share the experience of how I have been approaching my specific problem domain, and hope that it leads to some useful light-bulbs going off in someone else's head.

The Stocktake scenario is instructive because it tells us that Inventory Manager is directly involved in the stocktake procedure. It is not an idle bystander that accepts a bulk update at the end of the stocktake. The Stocktake GUI starts the process by marking all stock as outstanding. One by one items are ticked off from the list until the remainder must be considered shrinkage. This means that only one stocktake can be going on at a time, and that the work of stocktake might be undone if someone starts the procedure again. On the other hand, it means that a crash in the stocktake GUI won't lose us our current stocktake state. Does this division of responsibility strike the right balance? That's what this design process is intended to make us ask ourselves.

We decomposed the Shares function more than most others, and this gives us an opportunity to see how a diagram with many components might work. Note that I have more arrows now than I have labels against those arrows. I have been using a convention that continuing arrows carry the same data as the last one unless indicated otherwise. In this diagram we have the Buy or Sell Record moving about essentially untouched until it gets to General Ledger Export. Only then is it converted into a Financial Transaction for recording in the General Ledger.

I find it interesting to pinpoint components that have little to do with altering the content that passes through them. From a REST/web perspective we might apply the End to End principle here. Not only should the interim components not modify the data passing through them, they should generally not try to understand the data more than necessary either. If communication is in the form of documents, they should ignore but pass on attributes and elements even if they don't understand them. Take care to clean documents up at system boundaries in order to be able to assign blame, but otherwise try to keep things free and easy. Even the document clean-up should ideally be done by free standing document firewalls that are easy to reconfigure as the architecture evolves.

I haven't included all of the images this time around. If you would like to look at the whole (but still limited) set of diagrams I put together, point your StarUML here.

The Rest

Clearly 4+1 is not the whole story. There are levels of detail beyond what I have covered in my examples this time around, including interface control documentation and module design specifications. There are also a great deal of additional details that may accompany an architectural description to support design decisions made or to provide more detail.

Perhaps the most obvious gap in the 4+1 as applied at this level is a lack of data models for logical entities, for configuration data, and for interfaces. These data models can be extremely useful, and the earlier you have a stab at them the less likely you'll have multiple teams misunderstanding each other.

Other gaps include objectives for the decomposition, rationale for choices made, descriptions of how the architecture would change if current assumptions and constraints were to change, and a host of other detail. Any one of these could lead you down the path of discussing something to the point where you stop and say to yourself "Hang on. That doesn't make sense. What if...". Those moments are half the reason to put an architectural description together, and some of the value of the document can be measured in how many "oops"es are caught as compared to the effort you put into the document.

The other side of the document is obviously to steer the ship and record consensus decisions. Communicating the reasons for the current set of design decisions and the set of alternatives considered can therefore be crucial in the maintenance of the document over the project lifetime.

In the end, everything on the project is linked. From schedule to team structure to components to the list of change requests against those components. If you start modelling you might just end up modelling the whole world. Depending on the scale of the project it might be worth going that far. If you are working on something smaller you should be doing the minimum and assessing the rewards for taking any further steps with the document set.

Conclusion

I come from a fairly heavy-weight process end of the industry. It involves a Systems Engineering approach that has foundations in things like defense projects. The kinds of descriptions I have been talking about may not suit everyone, or perhaps even anyone. However, I hope they will make some of us take a step back and be aware of what we as a whole are building from time to time.

Again I am not an expert, and I hope my lack of expertise is not a hindrance or false signpost to the gentle reader. I am a software guy who has been walking a path lately between traditional software experience and the experience of those around me working on various projects.

I'll now return you to your usual programming of REST Architecture, and everything that goes with it.

previous in series

Benjamin

This is part four of my amateur coverage of the 4+1 architectural views. I left my coverage last time with the logical and process views complete. The logical view is a system's engineering coverage of the system's functional architecture. My Process View closed out most of the design aspects of the system. We now have a set of software components and functional interfaces which we may further refine.

The Development View and Deployment (Physical) view close out main set of views by moving software components out of their final assembled forms and into other contexts. The Development View captures components "in the factory". In other words, it shows build-time relationships between components. The Deployment View captures components being deployed to the operational system. If we were continue clockwise around the views through Development and Deployment we would naturally reach the Process View again. In other words, components once built and deployed are assembled into processes.

The Development View

The Development View is where I like to travel to in telling my architectural story, after the Process View. We have seen the final assembly of processes, now let's go back to where it all began. If your architecture has a strong distributed object focus this view will be relatively simple, but it does serve an important function.

The Development View is where we make a break from whole Logical Functions, and narrow ourselves down to their core for deployment. Let's follow our example through:

The packages view isn't very interesting. It would perhaps be more so if I had included actual database software or other basic components to get the job done. Those would very likely appear at this level, alongside packages we discovered in the Process View. There are no build-time relationships in this set, only run-time relationships.

The Development View for Sales includes all the components that made up the Process View diagram of the same name. These components are everything that is required to put together the Logical View Sales function.

Something has changed, though. This time around our packages are in the diagram itself. We can see a Sales package, and it clearly doesn't contain all of the software components involved. When we deploy "Sales" it will be this core set of components (only one in this case) that will be deployed. The Sales Manager component will be deployed as part of General Ledger. The Web Browser will be deployed as part of its own package.

Inventory shows multiple components within a functional core. Inventory contains both its client- and server- side components.

Shares shows that some of the run-time relationships found in the Process View are linked together at build-time. The Portfolio Manager controller component depends on and links together all of the other constituent parts.

The remainder of these diagrams are fairly boring. However, they do ask us to consider what libraries or other components we might have missed in our Process View analysis. This will be important contributors to the final set of software components and may significantly influence planning and estimation on a large monolithic project.

The Deployment (Physical) View

The Deployment View is about the final installation of components into the operational system. It is theoretically concerned both with the mapping of software components and processes to physical devices. However, again mine didn't quite turn out that way. I ended up focusing on packaging. Here is my top-level diagram:

This image shows the physical structure of final Accounts system. We have a Stock Handling workstation with a Scanning device attached. It contains the thick inventory client.

On the right we see a single Accounts server on which we plan to deploy the entire server workload. A generic client machine carries a Web Browser to render the various HMIs.

We are limited in what we can do with a UML deployment diagram like this. We obviously can't dive too far into the nature of the hardware components in what is really a software diagram. We can't draw the racks or get too far into the detail of the network architecture. These facets seem to fit outside of the 4+1 views.

The Deployment View for Sales is significantly slimmed down from previous logical-aligned views. For the first time we see the non-core components such as the Web Browser stripped away and absent. What is left is a pure server-side component, so we put it in a package of that name and deploy it onto the Accounts Server class of machine.

Inventory is more interesting. We have a client and server side to this functional core. They must be deployed separately. Even at this late stage we can see potential holes in our design: Did we think about the client/server split? Can we see components that need extreme bandwidth or latency settings, and should be co-located?

I haven't shown all the dependent components for Portfolio Manager under the Shares function core. I assume they are all included at build time and do not require special attention when it comes to this level of packaging.

Again, the rest of these diagrams are pretty boring due to lack of intelligent detail.

Conclusion

That concludes the main four views. I hope this coverage has been useful, and expect to see a number of "no, this is the way it is done!" posts in response. My approach has not been entirely kosher, and is not based on a broad enough base of experience. However, it does what I set out for it to do. It identified a set of software components and functional interfaces. It aligned the set through multiple views to the end user's functional requirements. It examined the set from multiple angles to see how they would play out:

• In their final assembled form,
• in the factory, and
• as packages during deployment.

It did not decompose components or interfaces any more than necessary, hopefully making the necessary high level decisions without boxing individual component authors in unnecessarily.

This approach is suitable for architectures of size, both REST and non-REST. Again, it might surprise some that REST has not specifically appeared at this level.

The way I view it is that this is an approach to selecting components to build and identifying the demands on interfaces between them. Each component will still be built according to best practice and will conform to appropriate standards. If I were to build a circuit breaker for the electric power industry, or an impulse fan for environmental control, I would supply digital inputs and outputs of a standard type for connections to RTUs, PLCs, or other field devices. If I am to build a software component, I will supply URLs to allow it to be connected freely and generally also. I will follow industry best practice for loose coupling, maximum scalability, and maximum evolvability. These views told me what to build. REST tells me how to build it.

To some extent I want REST advocates to see this other side of things. Whether you do this formally or informally and by what ever technique: There is a process of deciding what to build and what interfaces are required in software engineering. There is an architecture role that is about the specifics of what is needed in a particular system. REST is a technique that informs this level of architecture, but is subservient to it. I sometimes wonder if this is the main reason for disconnect between the REST and SOA camps. One is talking about which components to build, while the other is talking about how to connect them together. Little wonder that we can't agree on how to do "it".

I plan to round out this series with a discussion of the Scenarios View, and about other supporting documentation that might be needed to support your architectural decisions.

Just a reminder: You can find the StarUML model I used in this example here.

previous in series next in series

Benjamin

This is part three in a series that checkpoints my evolving understanding of The 4+1 View Model for architectural descriptions. I have already provided a description of the Logical View. This view captured End-user Functionality in a process based more deeply in Systems Engineering than in Software Engineering. Today I cover the Process View and its links into the Development View.

Let's revisit the set of views as a whole:

We can see the Logical View tracing independently to the Process and Development views. Each of these views then traces independently to Deployment. I had a lot of trouble figuring out how to make this work for me, from a UML purist and a tooling perspective. I finally settled on a slightly different approach based on software components.

I have ended up including the final set of components in each of the "design" views. The same set of components appear in the Process, Development, and Deployment (Physical) Views. This achieves the goal of relating different design views together. It also fits with the ideas in the UML Superstructure specification:

The component concept addresses the area of component-based development and component-based system structuring, where a component is modeled throughout the development life cycle and successively refined into deployment and run-time.

The process view I came up with is based around a UML component diagrams, where components from the upcoming development view are show assembled into running processes. This suits the kinds of designs I am doing. It may need tweaking to suit your own needs.

Let's dive into an example with a quick review the logical view functions:

I have elaborated on the logical view functions based on the process decomposition as follows:

The logical functions of Sales, Inventory, Shares and Tax Reporting are still present. New are the Web Browser and General Ledger packages. These packages are the result of design decisions in implementing each function. Let's look at Sales:

This diagram captures the whole Sales function, a theme I will follow throughout this and subsequent views. The constant relationship to the Logical View helps establish the link between requirements and all aspects of design. The Sales Ledger Updates interface is still present from the Logical View. This is served by the Sales Manager component in the identically named process. A Web Browser is used as part of the HMI, and General Ledger is used as part of the Historical Sales Register.

The Sales Ledger Updates interface is confronting right up front. Am I not a REST advocate? Isn't defining an interface like this against the principles of REST? Well, no. I fully expect this to be a REST interface. However, I am approaching this design from the customer focused logical perspective. From this angle, the most important thing to know is what information is transferred. An Interface Control Document will be required to identify the set of URLs that Sales Manager provides, and specify exactly which REST operations are necessary for each URL.

On the other hand, we could easily make the design decision at this point that the interface will not be based on REST. It could be a less constrained SOA. It is for this reason that I feel I can talk about REST being a constrained subset of SOA, part of the SOA family, or "SOA done right". At this point in the design process SOA and REST are indistinguishable.

The General Ledger is the next thing that jumps out. That wasn't in the logical view. There was no requirement for a General Ledger. Instead, Sales was supposed to make data available to Tax Reporting for processing. Here I have used my design experience to say that we should have a General Ledger (a summary record of any type of financial transaction). There is no point requiring the processes of Sales, Inventory and Shares to appear as part of the Tax Reporting function. The General Ledger allows us to put these functions at arm's length from each other.

The Web Browser is fairly obvious, but looking at Sales Manager: Why haven't we decomposed it further?

The answer is that in this case I want to give the developer of this service maximum flexibility in how they implement it. If there were a library that is common to other functions it would need to appear (which begs the question of why there are no such libraries... have we missed something?). Internal structure of a process that has no need for Intellectual Property separation or for other forms of separation might as well stay part of the same component.

Inventory follows a similar pattern to Sales. However, we do have two distinct GUIs identified the logical view. It makes sense to keep these separate, because they have quite different usage profiles. I have decided to use a thick client approach, here, rather than a Web Browser. Part of the overall Inventory HMI is a scanner, and the thick client grants me better access to the scanner's feature set than the browser would afford.

The obvious design question arising from this diagram is how Stocktake and Inventory Scanning GUIs coexist within the same process without any apparent coordination whatsoever. Have we missed a navigation GUI that allows us to select one mode or another? Should they simply be in different processes? Do they need a common library of any kind to communicate with Inventory Manager or drive the scanner?

Here I show the interior detail of a process. The Portfolio Manager is constrained by this design to follow a Model-View-Controller approach internally. The Views are fed to a Web Browser, or to the General Ledger by export. The Model is updated by its HMI, or from the Stock Quote Client driven by Periodic Execution. All of this is coordinated through the Portfolio Manager.

Tax Reporting extracts data from the General Ledger using a Reports Engine, and again uses the Web Browser for final display. The Tax Reports give me the opportunity to show a software component that is really simple configuration data. It is clearly separate from the Reports Engine, and uses a dependency relationship on the engine. The dependency direction could be viewed in either direction, however it is easier to trace requirements to a lower level with the arrow as stated. This diagram says that the Tax Reports use the Reports Engine to do their job, as opposed to the other way around. That means that we can talk about the Tax Reports as having to achieve certain GAAP requirements while leaving the Reports Engine itself with fairly generic requirements such as "must be able to query SQL".

The final images are fairly boring, just to show that the packages we discovered in our journey through the Process View will generally get their own diagrams and description. You could show the processes of components they connect to, but that information should already be present elsewhere. These non-functional packages will be flowed through the Development and Deployment (Physical) Views in due course.

You could argue that the inclusion of components the in above Process View means that there is really no design left for the other views. You would be right. The main objectives of this architectural description are met: To define the set of components and interfaces in the architecture. The subsequent views are relatively boring, compared to the exciting leap from systems-based logical to software-based process views. However, they each bring their own charm and provide useful checkpoints to discover flaws in your architecture.

I suppose another question-mark at this point is the detail of the internal interfaces between software components. I have identified links, but not tried to establish a functional baseline for these links. For this we would likely need to go through a round of requirements allocation and decomposition and follow the process of the logical view again. I defer the specific work on functional interfaces to the next level of design document down.

previous in series, next in series

Benjamin

I want to make clear at the outset that I am not an expert in 4+1. I have spent the last few months working with systems engineers on an Australian rail project, and this is the sum total of my systems engineering experience. I am reasonably happy with my understanding of both concepts as they apply to my specific situation, but this approach my not apply to your problem domain. I am using StarUML for my modelling, and this pushes the solution space in certain directions.

Overview

The first view of the 4+1 model for architectural descriptions is the Logical View. A classical computer science or software engineering background may not be a good preparation for work on this view. It has its roots more in a systems engineering approach. The purpose of this view is to elaborate on the set of requirements in a ways that encourages clear identification of logical functions, and also identifies interfaces to other systems or subsystems.

The general approach that I have been using for this view is a combination of data flow and robustness diagrams. The elements in these diagramming techniques are similar, but both are constrained in my approach to serve separate specific purposes. Robustness diagrams are drawn by working through an input requirements specification. Out of this process "functions" are discovered and added to a functional data flow diagram. Finally, a context diagram is constructed as an extraction from and elaboration on the function data flow diagram with external systems identified.

Interfaces are identified at every stage through this process, including Human-Machine Interfaces. The structure of robustness diagrams make it relatively simple to identify missing interfaces and other gaps in the analysis. The interfaces identified are reflected in the following Process View.

Context and Function Data Flow Diagrams

Data Flow Diagrams are very simple beasts, made simpler by the constraints I apply. For the purpose of identifying functions we use only circles for functions, and directed associations for data flows. Other features such as data sets or identified interfaces are banned in the function and context data flow diagrams. Data flows are labelled with the kind of data being transferred, and functions or systems are named.

Some guidelines for the definition of functions:

• An interface is not a function. You should not be able to draw a function that has the same name on an "in" data flow as it has on an "out" data flow.
• Configuration data and HMI are part of a function, not functions in their own right. We are setting out to describe the behaviour of the system in its completed state, not to design the system in this view.
• Non-functional requirements are implicit in this view. Don't try to draw these. This view may even help you distinguish between functional and non-functional requirements. Non-functional requirements and functions that are separate from their configuration data will want to be involved with every function and every control. Functional requirements, much less so. We are trying to come up with a set of functions and functional interfaces in order to maximise functional cohesion and minimise coupling. However, again we are not designing the system. The design in later views will follow the lines of these functions, and will elaborate upon them with solutions to non-functional problems.
• Functions are not use cases, nor necessarily collections of use cases. A use case can span multiple functions. A function is a cohesive set of requirements that is further elaborated in its robustness diagram or diagrams.
• The border between adjacent functions is always a data flow, never a control flow.

The set of functions should describe the system as the user sees it, rather than how it is built. The data flow diagrams describe relationships between systems or functions, not between software or hardware components. As additional constraints on these data flow diagrams I do not allow the expression of data stores at this level. They are described in the Robustness diagrams where necessary. This level is purely about flows of data between units of behaviour. The flows do not discriminate between REST-style, database-style, API-style, or any other style of interface. How the interface itself is expressed is a design decision.

Unfortunately, I was on crack when I wrote some of this. In particular, I have the concept of functions almost completely backwards. What I ended up calling functions are more like subsystems and... well.. you get it. I have since been corrected in a number of areas.

Here is how I would draw the robustness diagram today:

The context diagram remains the same, and I would continue to include it. I have started to prefer describing interfaces in terms of the information they communicate rather than trying to preemptively name and collate them based on other factors.

Here is a functions diagram, where I have collected the functions back down into a single figure. The key interfaces are still present, and I have two basic sets of data in the system. I have the ledger data, which is based on pure accounting principles. The other data is the historic share prices.

The share prices are acquired and stored based on live stock quotes. This information is combined with ledger-based accounts data for the shares investments in order to generate appropriate reports.

Other sources of data for accounts include the daily sales ledger acquired from the Point of Sale system, internal movement of warehouse stock, and input from stocktake operations.

Per-function Robustness Diagrams

Robustness diagrams consist of boundaries, entities and controls. Boundaries are interfaces to your system, including Human-Machine Interfaces. These are functional interfaces, so typically identify broadly the kind of information required or the kind of control exerted through the boundary. The set of boundaries may be refined over subsequent iterations of the logical view, but do not be concerned if they do not line up exactly with the set you expected. Multiple functional HMI boundaries may be combined into one user interface to provide the best user experience. Multiple software or protocol interface boundaries may be described in a single Interface Control Document.

The process of building up the functions involves reading through requirements one at a time. Look for vocabulary in the requirements that suggest a system boundary or entity. A single requirement may appear as an entity in your diagram, or many requirements may map to a single entity. Only a poorly-written requirement should require multiple controls to be associated with it. Entities are identified when one requirement talks about producing some kind of data to be used by another. An entity is a set of data, such as a database. We are not trying to build an Entity-Relationship model (I draw entity relationships outside of the main 4+1 views). Once the data set is identified it is not elaborated further in this view. An entity may be temporary or permanent data.

Some guidelines for the Robustness Diagram

• Controls bind everything together. Boundaries do not touch boundaries or entities. Entities do not touch entities. A control must be present between these end-points.
• Don't try to chain too many controls together. In general I would expect only a simple flow from one or more boundaries or entities through a control to one or more boundaries or entities. Requirements-mandated validation behaviour is a common exception to the rule, where the validation is subservient to a main control. It is especially useful to do this if the validation interacts with a different set of boundaries and entities to the main control. However, don't attempt to describe an algorithm by linking controls together.
• Don't go overboard with entities, controls, or boundaries. If your diagram has the same controls always operating on two different entities or boundaries, they may need to be collapsed into a single entity or boundary. If two similar controls are relating the same set of entities and boundaries, consider collapsing them into a combined control. The main goal is to establish how concepts from the requirements specification are related by behaviour. The requirements themselves are still there to describe the behaviour in detail.
• Be suspicious of a control that is not driven from anywhere, and entities that influence behaviour without obvious input. Look for missing functional HMIs and other unidentified system boundaries.
• Don't try to make every requirement a control. Many requirements are there to clarify or constrain a general functional behaviour. The goal of controls in these diagrams is to capture specific uses in order to bind the concepts held in boundaries and entities functionally together. We thereby discover the functionality required of known boundaries, as well as possible missing boundaries and implied behaviours.
• As entities are data, they are candidate points for splitting a large robustness diagram into multiple functions with a data flow between. However, avoid splitting cohesive controls apart unnecessarily. Try to describe a logical set of behaviour in a single diagram and as a single function.
• Robustness diagrams are not use cases, and are not collaboration diagrams. Directed associations can be useful in showing control flow from start entity or boundary through controls to their destination. However, ordering is not implied, nor is frequency or method of control flow. The level of granularity of these diagrams only allows us to say something like: "When data changes across this interface or in this entity, this kind of behaviour will happen involving these other boundaries and entities".
• Do not try to identify functions in the robustness diagrams themselves. Functions and controls are different concepts. Functions are behaviour with information exchange with the next function. Controls are behaviour that pass control flow to other controls and update entities or boundaries.
• Do not design internal structure into the system. Boundaries are system boundaries, not function boundaries. Entities are identified naturally as domain concepts out of requirements. They are not invented.

Example

The example I am using is of a very simple accounting system. I haven't written up formal requirements, but let us assume that the following model is based on a reasonable set. I will work though this example top-down, generally the opposite direction to the direction the model would have been constructed. The source model can be found here (gunzip before opening in StarUML).

The context diagram shows the Accounts system in context with an adjacent Point Of Sale System to acquire a daily sales ledger, and a broker system to provide live stock quotes. If I were presenting this in a formal document I would re-draw it in visio. Packages would be drawn as "function" circles, and interfaces would be removed (leaving only data flows). Since we are all friends, however, I'll leave it in the model form for the purpose of this entry.

Already we can see potential gaps. We get live stock quotes from the broker's system, but no buy or sell orders. Is this intentional, and the buy/sell orders are handled entirely within the Broker System or have we missed it from our analysis?

The set of identified functions tracks these external interfaces down to a lower level. The stock quotes from that broker system are being fed into a shares investing function. Sales receives the daily sales ledgers. Inventory is a self-contained function within the system. Tax Reporting uses data from all of these functions.

It is possible that a general ledger function could appear at this level, but so far in this analysis we have not determined the need for it. The system is used for shares investing, sales and inventory tracking. Any general ledger containing the combined records of all of these activities is a design decision at this stage. Tax reporting requires that we bring the data together in some form, but whether we mine it directly or convert for easy access remains unconstrained by this logical systems engineering process.

The Sales function has two main controls: To import the daily sales ledger from the POS System, and to generate reports. A historical sales ledger entity naturally appears out of this interplay, and becomes the boundary between this function and the Tax Reporting function. We discover a HMI is required to request report generation. Does this HMI need to do anything else? Is printing required? If so, do we need to be able to cancel print jobs? Are these functions of the system?

The Inventory Function has four controls. Two are relating to scanning stock in and out of stores. Another two are related to stock-take. Stock is re-scanned and compared against Inventory. Once all stock is scanned, the Stocktake HMI accounts for any shrinkage. Both the Inventory Scanning and Stocktake HMIs can take a number of forms. There might be keyboard and VDU entry. With any luck there will be a bar code or rfid scanner in the mix. The same scanner might be used as part of both HMIs, and both HMIs may be combined into the one end user HMI.

Investment Tracking involves recording of purchase and sales of stocks. We can run reports based on the book value of these stocks (ie, how much we paid for them), or based on their current market value. In order to do the latter we need to keep a historical record of stock prices. Maintenance of this record has a clear entity boundary at Historical Stock Prices that makes it a candidate for separation into a new function. We still might do that if this function becomes unruly. I have placed a kind of stub interface in here for periodic execution of stock quote updates. However, this obviously needs more thought. Where is the HMI to control how often these updates occur? Can they be triggered manually?

I foreshadowed that the entities from other functions are referred to directly in the Tax Reporting robustness diagram, and here it is so. This does not directly imply that the Tax Reporting function has access to the databases of the other functions. It simply means that the set of information (or relevant parts of that set) is transferred to the tax reporting function at appropriate times. This could be by direct database access, by an ETL into the tax reporting function's own database, by a RESTful data exchange, or by some other means.

And this is how I would do it, now. This is a very basic diagram, but you can see that it is software centric in that it balances data with functionality, and views the world as a set of classes and objects. I would generally start with an entity-relationship or a class diagram relating to domain-specific vocabulary from the requirements specification, and work from there.

To some extent I find this diagramming technique freeing. I don't have to worry about the borderline between software and systems engineering. I don't have to worry about components. I can just draw as I might have in my youth. It will feel familiar to software developers, and a software developer should be able to judge whether or not it works to convey the appropriate information.

Conclusion

The Logical View is somewhat airy-fairy, and the temptation is always there to introduce elements of the design. Resist this temptation and you should end up with a description that clearly separates constraints on your design and design decisions you have made. The set of interfaces your system has with other systems is a real constraint, so all of these interfaces appear here in some form and behaviour required of those interfaces is identified. You may need to round trip with the process view (and other views) in order to fully finalise this view.

It is likely that the Logical View (especially its robustness diagrams) will identify some inconsistency in vocabulary in your source requirements. It is worth putting at least a draft of the design together before finalising requirements for any development cycle.

previous in series, next in series

Benjamin

I have been working lately on a number of architectural descriptions, and have been using the 4+1 style as part of these descriptions. This has been an interesting experience for me, because I have previously worked more in the technical sphere. These descriptions remain quite abstract, to the point that they end up exposing a level of Service-Orientation (or perhaps just component-orientation) but do not reveal that REST is the architectural style used for implementation.

The 4+1 style was originally proposed by Philippe Kruchten of Rational (and later IBM) fame. 4+1 is consistent with IEEE 1471's advice that architectural descriptions be broken up into multiple views. The principle is that it is impractical and confusing to use one diagramming technique or one style of description in summarising an architecture. Multiple views are required to meet the needs of different stakeholders.

Philippe's 4+1 main views are:

• Logical,
• Development,
• Process, and
• Physical (also known as Deployment, to avoid confusion with the term Physical Configuration Audit)

These views all typically describe static arrangements of elements. The +1 view is Scenarios, allowing for the demonstration of example modes or uses of the architecture in a more dynamic view. The Logical View is all about requirements analysis. This is the view for exploring and expanding on this design's requirements baseline. The other views provide a narrative that relates this requirements analysis to the final design, and views that design from a number of different perspectives.

The main goal of my architectural descriptions has been to come up with a well-formed list of components and interfaces for further expansion in other documents. In my case components are either configuration data, libraries, or application software. Interfaces could be internal, external, or human/machine. To this end, I have defined a set of components across the design views to show how they appear in terms of their runtime, build-time, and deployment-time relationships. I have used the functional decomposition of the Logical View to guide and partition the design through the other views.

The Logical View covers the functional requirements of the system, and is closely related to the approach of Systems Engineers to the problem of design. The Process View is intended to show how threading and multiplicity work to achieve non-functional requirements. I use this view as the "main" design, showing run-time relationships. Build-time relationships appear in the development view, and essentially model the Configuration Management environment in which the software is developed. I have used the final Deployment View to plan how components are packaged up for deployment onto particular classes of machine. This view can also be used to show a final (but abstract) view of how target hardware is connected together in service.

There isn't a great deal of information on 4+1 out there on the Web, so I plan to produce a series of articles covering the different views and diagramming techniques I have personally been employing. I have adapted 4+1 slightly to my own needs, and where I have knowingly done this I will attempt to distinguish between the "real" 4+1 and my own tweaks.

Next in series

Benjamin

Regular readers of my column will be familiar with a common theme: REST provides improved decoupling, evolvability and serendipitous reuse over unconstrained SOA. What you may not expect is a break from dense blocks of text into a world of diagrams (or at least a breaking up of dense blocks of text with diagrams). This article is on REST fundamentals. I'll start with a couple of version of the REST Triangle diagram I drew for Wikipedia a few years ago.

Dissecting a REST Triangle Instance

This diagram has a URL on the top point of the REST triangle, the URL of my blog. The left-hand points on the base refers to the HTTP operations GET, PUT and DELETE. The right-hand refers to various kinds of document that can be transported by HTTP: HTML, Plain Text, VJournal, and Atom.

You might notice while looking at this diagram that context decreases in clockwise direction around the triangle. Most of the meaning of an operation is held in its URL. You don't update someone else's blog when you PUT to my URL. You update mine. You won't retrieve stock quotes when you GET from my URL. You will only see my latest musings.

The types of transported documents are much more general than a single URL at my site. In fact, you can exchange HTML or Atom documents with a range of sites of differing natures around the World-Wide Web. However, there is a clear difference between HTML and Atom in terms of context. You will find Atom documents or documents expressible as atom at far fewer URLs than you will find documents expressible as HTML.

HTML is a low-jargon document type that allows text-intensive data to be rendered by a browser irrespective of its topic or context. It minimises coupling between client and server by avoiding being specific to a given purpose. However, in doing so it also gives up potential semantic riches. Atom is able to convey more interesting machine-readable semantics at the cost of being applicable to a smaller set of documents.

This is not an accident of either HTML or of Atom. I see a general case of friction between transferring high value semantics using specific vocabulary and jargonising communication so that it can only be understood within a restricted context. High value semantics generally increase coupling, and the only way to deal with this friction is to work the standards bodies hard to figure out what the world can and simply can't agree on. Matching vocabulary to context in a way that minimises jargon is the main practical challenge of any Semantic Web, a World-Wide Web that contains significant quantities of machine-usable information. Don't believe anyone that says building any kind of meta-model for data will make the semantic web happen. That isn't solving the problem.

The least content of all is held in HyperText Transfer Protocol operations such as GET. It simply means "transfer a document to me of a type I say I accept from the URL I specify". The Accept header and URL are what ensure useful data is transferred and that it can be parsed and interpreted by the client, becoming information. The GET operation is just there to make the bits move.

Generalising the REST Triangle

This second triangle diagram elaborates on and generalises the first. URLs are at the top providing context. Document Types are further clockwise; sympathetic to the context but minimising exposure to it in order to minimise coupling. Protocol Operations are furthest around the clock, and give only the barest nod to context.

The important thing about Document Types and Protocol Operations for what I am going to talk about next is that they are both constrained. If you don't control or can't list the methods in use within your system you are not doing REST yet. If you don't control or can't list the document types in use within your system you are not doing REST yet.

That second point can easily take people by surprise, especially when combined with a warning against "application/xml" or "application/rdf+xml" document types. A document type defines the information schema of a document and how that information is encoded as data. "application/xml" doesn't do that. I can give your application an XML document that contains exactly the information that it needs, but unless it is encoded in a way your application understands it remains simply data. If we are going to get serendipitous reuse you must understand my encodings whenever I transfer information to you that you should understand. We do that by controlling the set of document types in use and making sure that there aren't different encodings emerging gratuitously for the same thing. The other side of that coin is that whenever an application accepts a particular kind of information it should support all known encodings of that information.

So where do we go from here? The answer is REST Rewiring:

Building an Information Pipeline

Here I show a GET and a PUT operation. Both transfer data at the client's request and on the client's authority, but do so in different directions. A GET transfers a document from URL (however that might be served) to Client. A PUT transfers a document from Client to a URL (however that might be interpreted). Any URL can deal with a GET or PUT if that operation is meaningful enough to have been implemented, and can reject the request with a reasonable response code otherwise. Stabilisation of these main document pipelines between clients and servers is the first prerequisite to most of the useful properties of REST.

The next thing to look at is what kinds of documents the pipelines are transferring. Remember our earlier discussion: The set of document types should be controlled such that the recipient understands that document type understands any information that could be meaningful to it. So not only is the document pipeline well understood, but the documents being transferred can be converted to information by any recipient for whom it is meaningful to have been sent the document.

By constraining Protocol Operations and Document Types we have produced an information pipeline, so what's left? Only to connect it to a specific URL.

Connecting up an Information Pipeline

The information pipeline still has limited meaning outside of its context, and the URL provides this missing context. The client issued a GET request to a given URL, not because they could, but because a user wanted it to happen. Somewhere in the mix a user is configuring or driving a client to connect its information pipelines to specific URLs for information exchange. What is done with the information ultimately depends on the design of the recipient, which in turn depends on what the author and user decided the information was for.

Not to get too philosophical about it, but: The URL rounds out the meaning of the information transfer. The other aspects of communication, those making up the actual information pipeline, are relatively independent of the meaning of the transfer. More importantly, they don't get in the way when the user decides one day to connect the information pipeline to a different URL. Serendipitous reuse is the ability to reconnect information pipelines without gratuitous protocol differences getting in the way.

Getting out of the way of Communication

REST makes it the responsibility of the developer/architect to control the sets of methods and document types in use within an architecture. By doing so it makes life initially harder for the developer. They have to think about how to deliver services though the uniform interface of their architecture. Once thought though, serendipitous reuse follows (funny how serendipity always seems to follow hard work...).

This may seem like a fault in REST thinking, however look back to the problem of the Semantic Web. It isn't a technical problem. It can't be solved technically. The problem is not the technology, it is the lack of agreement between developers. So long as we are continuing to create different document types to convey the same or similar information there is no hope for machines to map one to the other quickly enough or accurately enough to match the expansion. What is needed is for the technology to move appropriate parts of the socio-technical problem back into the social space.

REST principles of are a socio-technical answer to building a Semantic Web, possibly the only one known to build a broad base for information exchange so far at scale.

Pipeline Evolution Paths

The World-Wide-Web doesn't stop at producing a socio-technical solution for building a static system. It goes on to provide evolution mechanisms for Protocol Operations (not implemented response codes, must-ignore headers, etc), for Document Types (must-ignore for unknown XML elements and attributes, content negotiation), and even for URLs (redirection). These facilities round out the solution in terms of the information architecture, providing long term evolution mechanisms.

Benjamin

Reliable messaging based on idempotency allows clients to retry requests safely when they time out. Another property of PUT and DELETE requests is that intermediate requests to a given URL can be discarded. Only the last request needs to make it through. This model allows application-level guarantees to take the place of transport layer guarantees to achieve high performance and scalable reliability. To a network purist this may seem like a step backwards, but in practice it is an example of better training of and agreement between developers leading to a simpler and more sustainable architecture.

The guarantees of PUT and DELETE must be applied consistently in order for APIs to make use of these properties to provide reliable messaging, and that depends a lot on the URL to which they are applied. I have a couple of examples from the SCADA world to share in this article that may be instructive elsewhere. The first relates to URLs that demarcate a changing scope of state, and the second is based on the problems of a mixed REST/non-REST architecture.

URLs with a changing scope

SCADA systems often have a centralised list of alarms that operators are expected to respond to quickly. Design of these alarm lists vary, and sometimes for particular reasons it is useful to request the system acknowledge or erase all alarms in the list.

A naive approach would be to model the requests thusly:

Acknowledge all alarms

PUT https://example.com/all-alarms/acknowledged (text/plain, "1")

Erase all alarms

DELETE https://example.com/all-alarms

The problem with this approach comes about when we consider the effect of repeating a PUT. Is it safe to repeat?

The answer in this case is "no". It is not safe to repeat, because a subsequent PUT or DELETE will affect a different set of alarms to the first request. The first request could be successful, then new alarms raised between the first request and the second. When the second request arrives, it is not "safe". It has an effect greater than a GET request.

This can be a fuzzy line to draw. Two clients making requests to the same URL will naturally interleave. It will always be the last valid client to successfully make or repeat its request that ends up winning out in the race condition. Client A wins out when A makes a request, then B, then A repeats. In this case, A's second PUT is not safe exactly because its request will negate the effect of B's request. However, this is a genuine race of intent. Clients A and B are in a race to see which of their intents will take effect.

This kind of race can be avoided by narrowing the scope of the URLs in use. A better approach would be as follows:

Acknowledge all alarms

PUT https://example.com/all-alarms/acknowledged?before=2008-03-07T12:32Z (text/plain, "1")

Erase all alarms

DELETE https://example.com/all-alarms?before=2008-03-07T12:32Z

These URLs demarcate the same scope each time the request is repeated. Alarms raised after the specified time are not affected by repeated requests, making the repeats safe.

Mixed Idempotency Environments

Consider the case of a circuit breaker in a power distribution system. These devices can be massive, and some have to be replaced after only a small number of trips. Think of a mechanical lever. Each position (tripped or closed) has a separate digital read-back. (Tripped=true, Closed=false) means that the lever is in the tripped position. (Tripped=false, Closed=true) means that the lever is in the closed position. (Tripped=false, Closed=false) means that the lever is in motion, or sometimes that the lever is stuck. (Tripped=true, Closed=true) should be an invalid state.

Sometimes the read-backs can be faulty. Even a breaker that reads as being in the closed state might not actually be conducting electricity. An operator can typically tell this by reading the voltage back from various meters around the network. In this case, the user knows more about what is going on than the machine does. So, how do we get an apparently-closed breaker really into the closed position? We given the gyros another kick. We send another request to close.

Is this idempotent? Can it be modelled as a PUT?

The answer is "no", or at least... not cleanly. A PUT of (text/plain, "1") to indicate a close would not be appropriate if we expected a repeat of the same PUT to kick the gyros again. Likewise, it is not appropriate for the system to automatically retry such a request. The expense in terms of wear and tear may be too costly. What is needed is a different kind of request; a non-idempotent one. This is best to model as either a POST or as a domain-specific method.

There are various ways to ensure reliability in this situation. The simplest is to involve a human in the loop. We specifically don't want to repeat our request due to a desire to keep a human involved with any costly decisions. Avoiding automatic repeat of non-idempotent requests allows us to simply return an error to the user. The user can reassess the situation, and determine for themselves whether to give things another kick along.

Other reliability approaches are possible, but not as strong. It is possible to arm an operation, then execute it as two separate requests. Either the arming or execution can be repeated individually without reissuing the same request. Another alternative is to move to a complete WS-style reliability model.

These sorts of cases don't really come up as part of interactions between information systems. It isn't until you get into the physical world that you really run into this problem. My suggestion is that idempotency is certainly king. Putting a human into the loop for other request that they must be involved in anyway seem reasonable for the cases where reliability is a problem, ie during failover of typically reliable control system.

Conclusion

PUT and DELETE requests are all about specifying the expected outcome of an operation, and allowing the server to decide how to achieve the outcome. Consider the effect of an automated system either discarding intermediate request, or repeating requests when considering whether your PUT or DELETE request on a given URL is appropriate. If it doesn't make sense, you are probably not specifying the URL or method correctly.

Benjamin

MIME is the corner stone of document type identification on the Web today. XHTML, for example carries the application/xhtml+xml media type. On the other hand, it also carries a https://www.w3.org/1999/xhtml XML namespace. Media types are controlled centrally by the IANA. XML namespaces are URLs, and therefore open to anyone to create. What are the trade-offs, and what is the future for document type identification?

Decentralisation

Mark Baker comments on recent attempts to decentralise media type assignment, saying:

Been there, tried that. I used to think that was a good idea, but no longer do.

I'm a centrist on most technical and political issues. I see both sides of this debate. On the one hand, tight technical control ensures that document types on the Web are well controlled. This is a good thing. It means that different components of the Web's architecture can exchange information through these data types. On the other hand, there are always going to be document types that make sense outside of the context of the Web. These might be used within an isolated railway control system, or in a Business-to-Business pairing or grouping.

As the importance of the REST architectural style sinks in outside the Web it becomes less likely that the set of Web content types will be sufficient to convey all useful semantics. If we make the assumption that all document types ultimately need to be identified, we might to ask questions like: "What is the content type of the configuration file format for my SMTP server?". As we spiral outwards from this kind of very specific case, "What is the content type of a machine-readable description of a Railway?". Neither of these cases appear on the Web, and the IETF is unlikely to be an appropriate forum to discuss standardising the identification of these documents. Web protocols and standards extend past the single World-Wide Web, and to some extent I think this is important to recognise in the development of these standards.

MIME vs URL

Even if we do open up document types to a very wide base, MIME Types and URLs do not contain the same information. Importantly, URLs are generally opaque. In contrast, MIME Types can be interpreted. An atom document containing future xml content knows from the application/future+xml type attribute that it is contained as an unescaped XML sub-document. It can interpret a text/future as meaning that the sub-document is an XML text node. It can interpret the absence of these conditions as meaning that the content is binary, and is base64 encoded. Likewise, parameters on text document types can indicate information such as character encodings.

A danger in heading into the URL approach for document identification is that we loose this additional metadata. We would ideally be able to extract from a document both its type, and any parent types it may have that we could understand.

Where I stand

My solution for the moment is to just make up media types for special-purpose applications. These types are not registered with the IANA, and are not exchanged in general parlance. My theory is that a time will come where this type either comes into contact with another type with the same name, or another type with the same essential data schema. There will be some kind of conflict when either of these things happen, and the conflict will have to be resolved through social (rather than technical) means.

Once these types are well-enough developed for this kind of conflict to have occurred, it is likely that they will be ready for inclusion in some form of register. That might eventually be the IANA, but I suspect that satellite bodies will need to participate in the control of the document type space.

The question with this approach is where it leaves us with XML namespaces, and with URLs in general for document type identification. At present I don't recommend the use of XML namespaces at all. I think that MIME Types are king, and will remain king for the immediately forseeable future. XML namespaces should therefore be ignored by consumers, in general as redundant information. On the other hand, I might just be swimming against the tide on that one. I guess that atom could get by perfectly well with an XML namespace and no type attribute for XML sub-documents. Perhaps there is no practical benefit from being able to parse a document type for additional metadata.

Benjamin

Tim Bray brings up the issue of what happens to private web sites after we die. It may be self-important of me, but I think the state has some responsibility for preserving the public works of its citizens. I first wrote about this problem back in January 2007.

My theory is that digital works such as personal web sites should be considered analogous to other published works, such as books. Libraries should be responsible for performing restoration work on the data by moving it to their servers, and by maintaining it and its associated domain registrations thereafter. The cost of maintaining this store in terms of storage and bandwidth should be easily minimised. It seems therefore reasonable that the state should attempt to preserve all the published digital works of its citizens.

On the other hand, perhaps we vanity bloggers would all be better of moving to hosting on free public sites hosted by existing large companies. If the private sector is already meeting the need, there is little chance that government will step in to stop the rot.

Benjamin

Rusty,

I agree that the SQLITE_BUSY return code is insane. The root cause is that sqlite is not using blocking operating system locks, meaning it has to poll to determine whether it can obtain access or not. Internally it has already retried its locks a number of times before reporting SQLITE_BUSY back to you.

I hacked sqlite 2 for WRSA's internal use to use blocking locks. Unfortunately, I have never gotten around to figuring out whether blocking locks can be introduced to v3 without causing problems. The v3 locking model is much more complex. Most of this feature is held in a single source file (os.c, iirc), so it should be possible for a single human to get their head around it. If anyone does get it together, perhaps it would be worth submitting a patch back to DRH via the mailing list.

Benjamin

I have had a rough time with services of late, but am now officially back on the Internet. To celebrate, I have done some minor revision to a RESTWiki article on how to design XML file formats for semantic-web-scale information exchange. I have also summarised some recent blog content into a page on how to do reliable messaging with plain HTTP and simple idempotent messages.

Have fun reading!

Update 2008-02-09: And here is another article on achieving high availability characteristics between a HTTP server and client

Benjamin

Embedded systems have to work under adverse conditions without human supervision. Embedded components that participate in distributed systems need to work despite arbitrary network delays, and with periods under which available bandwidth is extremely low. In these environments it is important to simplify communication to the basics, and these basics closely align to REST messaging on the Web.

REST is about designing interfaces around state transfer, either to or from a resource using standard methods, and in a standard format. Consider the case of an online jukebox. A client can sample the state of https://jukebox.example.com/current-track/title with a GET request. Having received the response in (say) a text/plain format with "Ring of Fire" as the value, the client is able to interpret the result or present it to a user. A subscription relationship could provide follow-up representations as the track changes over time.

A PUT (or DELETE) request transfers information in the other direction. Instead of simply capturing the state of a particular variable, it also captures client intent. An authenticated and authorised client PUT to https://jukebox.example.com/playing of text/plain "0" may pause the jukebox. The client may PUT "1" or "0" to the URL as its intent changes through time from paused to playing and back again.

All networks are to some extent unreliable. Packets get lost due to congestion or other factors, and this translates to delays and constrained bandwidth over the TCP layer. Reliable embedded systems cannot assume that they will be able to transfer every state they intended. While we can assume that any individual message will arrive eventually at its destination, a sequence of messages that grows at a rate faster than available effective bandwidth may never be fully delivered. Worse still, even temporary poor bandwidth conditions can trigger excessive delays in conveying a server's most recent state or a client's most recent intent if messages sitting in the queue before the most recent state or intent must be processed first.

The nature of a PUT or DELETE request is that previous resource state is obliterated. This poses a curious solution to the poor bandwidth problem, where intermediate intents are simply dropped. Consider an indecisive user who clicks "pause", then "play", then "pause" again. The intermediate "play" request can be completely discarded if it failed to get through quickly. The client can keep a buffer of only the latest state it intends for the variable, only sending and retrying that state. This can be implemented in generic library support code based on an understanding of the generic PUT request's semantics.

A subscription mechanism based on transfer resource state can exploit the same feature: The state of the resource being monitored can change more rapidly than the available effective bandwidth. While a client may benefit from seeing intermediate states, the only state it must see is often the most recent one.

Both changes of intent and changes of state can become more complex when resource state overlaps. For example, it may not be obvious that a PUT to <https://jukebox.example.com/currentSelection> should extinguish an outstanding request to pause the jukebox. Say the selection of a new track automatically starts the jukebox playing from a pause. This is a new intent that should extinguish the old, including any PUT request that the client has so far failed to send.

The simple solution to overlapping resources is to split these resources so that they don't overlap: Require the client to issue both a PUT to https://jukebox.example.com/currentSelection and https://jukebox.example.com/playing in order to achieve the desired compound effect.

This is not to say that our PUT requests cannot have side-effects affecting other resources. For example, the submission of any purchase order will affect accounting records somewhere. Every request is likely to have an affect on server logs, and these too may be accessible as resources.

The distinction between overlapping resources that make intent difficult to isolate and those that are merely normal side-effects is in the methods those additional resources support. It is only intent we should be disambiguating. We should be attempting to ensure that every intent is only conveyed through one resource: That multiple resources don't carry overlapping intents by their PUT requests. It is still ok that resources without PUT implementations overlap.

Clients that send requests with overlapping states need to take additional care when it isn't possible to avoid overlap of intent. In these cases the client must avoid issuing simultaneous requests to the overlapping resources.

REST's standard methods for transferring state and intent provide a platform in which generic code can identify and discard intermediate intent. Designing a system around REST methods thereby improves response times for conveying recent state and intent, along with its other benefits. The concept of conveying intent through PUT requests also influences how we design potentially overlapping resources in our architectures.

Benjamin