Sound advice - blog

Tales from the homeworld

My current feeds

Sun, 2006-Nov-05

REST, SOA, and Interface Generality

Mark Baker recently wrote about about the difference between the best SOA pratices of today, and REST. I find it difficult to pin down authorative statements as to how the SOA architectural style is defined or best praticed, however Mark sees a gradual conformance to REST principles. He described the main outstanding issue as "generality". I think my considered answer would be "decoupling for evolvability".

I think the answer is a little more sophisticated and involved than "generality". As Stu Charlton notes:

Even old crusty component software had this concept (operations constrained & governed to maintain relevance across more than one agency) - Microsoft COM specified a slew of well-known interfaces in a variety of technical domains that you had to implement to be useful to others.

I made a similar note in the current :

Uniform interfaces reduce the cost of client software by ensuring it is only written once, rather than once per application it has to deal with. Both REST and RPC designs may try to maximise the uniformity of the interface they expose by conforming to industry or global standards. In the RPC model these standards are primarily in the form of standard type definitions and standard choreography. In REST it is primarily the choice of standard content types and verbs that controls uniformity.

I think the real answer is held in how the REST Triangle of nouns, verbs, and content types allows different components of a protocol to evolve separately towards uniformity. We think of HTTP as a protocol, but it clearly does not capture the whole semantics of communication over the web. The full protocol includes a naming scheme, a set of verbs, a set of content types, and a transport protocol such as HTTP.

The REST Triangle of nouns, verbs, and content types

In current REST practice the accepted naming scheme is the URL, and the accepted verbs are basically GET, PUT, POST, DELETE. The current content types include html, svg, etc. However, Fielding's disseration defines none of these things. They are meant to each evolve independently over time, never reaching a time when the evolving architecture is "done".

If we contrast the REST view to classical base-class, arbitrary method, arbitrary parameter list object-orientation we can see a few differences. I think the main difference is that a classical base-class defines both the verbs of communication and the content types in one coupled interface. The REST approach is to determine separately which verbs are needed for universal communication and which content types are needed. This decoupling allows someone needing new semantics to be transferred between machines a significant lead up to defining the whole necessary protocol:

The naming system is well-defined and well-understood. No need to do anything there. The standard verbs are sufficient to operate any any virtualised state. Easy. So the only problem left is the definition of a content type to carry the semantics as a virtualisation of the state of your objects.

In REST, the protocol for delivering HTML documents to a user is closely related to the protocol for delivering images. The different protocols for delivering images (uri+verbs+svg, uri+verbs+png) are also closely related. Retaining the meaning of a verb between different content types and retaining the meaning of a content type across different verbs is a big leg up when it comes to innovation in the communications space.

Our software cannot communicate semantics unless our programmers agree with each other. This is an intensively social process involing politics of money, power, greed, personality, and even occasionally some technical issues. However, REST is about not having to start from scratch.

REST is about starting from a shared understanding of the noun-space, a shared understanding of the verb space, and a shared understanding of the content space. Whichever one of those has to evolve to support the new communication can be targetted specifically. It is easier to agree on what a particular document format should look like within the constraints of the classical verbs. It is easier to agree on how publish-subscribe semantics should be introduced within the constraints of the classical content types.

Stu also writes:

[U]niformity, as a technological constraint, is only possible in the context of social, poltiical, and economic circumstances. It's an ideal that so far is only is achievable in technological domains. HTTP, while applicable to a broad set of use cases, does not cover a significant number of other use cases that are critical to businesses. And HTTP over-relies on POST, which really pushes operational semantics into the content type, a requirement that is not in anybody's interest. In practice, we must identify business-relevant operations, constrain and govern interfaces to the extent that's possible in the current business, industry, and social circumstances, and attempt to map them to governed operations & identifiers -- whatever identifiers & operations are "standard enough" for your intended audience and scale.

And perhaps this is a point of confusion at the borderline of REST practice. REST advocates seem almost maniacal about uniformity and the use of standard methods and content types. I have highlighted lack of standard content-type usage in otherwise very precisely RESTful designs. REST has a hard borderline defined by its architectural constraints, and anything that falls outside of that borderline can be seen in black and white as simply "unRESTful". However it would be a fallicy to suggest that all software SHOULD be RESTful.

Most software on the ground will use a fair number of non-standard content types. There are simply too many local conventions that are not shared by the whole worldwide community for every exchanged document to become a standard. Applying as many RESTful principles as makes sense for your application is the right way to do it. As Roy pointed out in a recent rest-discuss post:

All of the REST constraints exist for a reason. The first thing you should do is determine what that reason is, and then see if it still applies to the system being designed. Use the old noggin -- that is the point.

REST is very strictly an architectural style for hypermedia systems crossing the scale of the Internet both on a technical and social level that allows for the construction of generic software components, in particular the generic browser. If your application does not involve developing a browser or your architecture is smaller than the scale of the Internet there are bound to be constraints that do not apply. You will still benefit by following the constraints that it makes sense to follow.

I find that following the constraints of a controlled set of verbs and content types over a uniform interface is extremely valuable even in a small software development group. Naturally, our set of controlled content types and even verbs does not exactly match up to those common on the web. Local conventions rule. However applying REST principles does make it easier to agree on what communication between our applications should look like.

It does make it possible to write a generic data browser. As local content types became stategically important we tend to look towards the wider organisation as for standardisation, and we sometimes aspire to standardation outside of the company and corporation. You could argue that until we reach that end-point we are only being nearly-RESTful rather than RESTful, but the benefits are there all the way along the curve. Perhaps the correct interpretation of Roy's use of the word "standard" in his thesis is that the verbs and content types are understood as widely as they need to be. This scope would typically differ between the verbs and content types. Agreement on the use and semantics of GET is required by a wider audience than is agreement on the use and semantics of svg.

Benjamin

Fri, 2006-Nov-03

Introducing SENA

A am currently in the process of authoring an relating to an internet-scale mechanism. The protocol is HTTP-based, and REST-like. I am currently denoting it as the Scalable Event Notification Architecture (SENA), a play on the old (GENA). It is intended for time-sensitive applications like SCADA and the relaying of stock market quotes, rather than for general use across the Internet. The draft draft can be found here and remains for the present time a work in progress.

Feedback Welcome

I have been soliciting targetted feedback on the document for the last few weeks, with some good input from a number of sources. Please consider proving feedback yourself by emailing me at benjamincarlyle at optusnet.com.au with the subject line "HTTP Subscription".

Architecture

The architecture consists of an originating resource, a set of clients, and of intemeditaries. Clients use a new SUBSCRIBE HTTP verb to request a subscription. This may be passed through without inspection by intermediataries, in which case the server will answer the request directly. A subscription resource is created by the origin server which may be DELETEd to terminate the subscription. A notify client associated with the subscription resource sends HTTP requests back to a specified client Call-Back URL whenever the originating resource changes.

Instead of passing the requests through directly, intermediataries can participate in the subscription. They can intercept the same subscription made by several clients themselves, and subscribe only once to the originating resource. The intermediatary can specify its own notify resource which will in turn notify its clients. This has a similar scalability effect to caching proxies on the Web of today.

Notification Verbs

I currently have a number of notification request verbs listed in the draft. The simplest one that could possibly work is EXPIRE. Based on the EXPIRE semantics, a NOTIFY resource could be told that it needs to revalidate its cache entry for the originating resource. If it is a client the likely response will be to issue an immediate GET request. This request will have a max-age:0 header to ensure revalidation occurs through non-subscription-aware intermediataries.

If the notify resource is operating on behalf of an intermediatary, it may choose to fetch the data immediately given that clients are likely to ask for it again very soon. Alternatively, it may wait for the first GET request from its clients to come in. Because it has a subscrpition to the originating resource, the intermediatary can safely ignore the max-age header. This allows the intermeditary to perform one revalidation for each received EXPIRE request, regardless of the number of clients it has.

The good thing about EXPIRE is that its semantics are so weak it is almost completely unnecessary to validate that it is genunine request from the real notify source. The worst thing an attacker could do is chew up a little extra bandwidth, and that could be detected when the originating resource consistently validated the old content rather than having new content available. EXPIRE also allows all normal GET semantics to apply, including content negotiation. The main bad things about EXPIRE are that it takes an extra two network traversals to get the subscribed data after receiving the request (GET request, GET response), and that you really have to GET the whole resource. There is no means of observing just the changes to a list of 100,000 alarms.

The alternative system is a combination of NOTIFY and PATCHNOTIFY requests. These requests carry the state of the originating resource and changes to that state respectively. The big problems with these requests are in their increased semantic importance. You must be able to trust the sender of the data, which means you need digital signatures guaranteed by a shared certificate authority. This introuduces a significantly higher processing cost to communications. Useful semantics of GET such as content negotiation also disappear. I am almost resigning myself to these methods not being useful. They aren't the simplest thing that could possibly work.

Summarisation

One of the explicit features of the specification is summarisation of data. Most subscription models don't seem to have a way of dealing with notifications through intermediataries that have fixed buffer sizes to a set of clients with different connection characteristics. If an intermeditary has a buffer size of one and recieves an update that can only be delivered to one of two clients, then recieves another update... what does it do?

The intermediatary usually either has to block waiting for the slowest client or kick the slowest client off so that it can deliver messages to the faster clients. The third option is to discard old messages that have not been transmitted to the slow client. In SCADA-like situations this is an obvious winner. The newer message will have newer state, so the slow client is always getting fresh data despite not getting all the data. The fast client is not stopped from recieving messages, so gets any advantages their faster connectivity is designed to bring. Most message delivery mechanisms don't know and can't take the semantics of the message into account. SENA is explictly a state-transfer protocol, thus these semantics of "old, unimportant state" and "new, important state" can be taken into consideration. PATCHNOTIFY reqeusts can even be merged into each other to form a single coherent update.

The EXPIRE request can also be summarised. New EXPIRE requests trump old requests. There is no point in delivering multiple queued EXPIREs. Likewise, the data fetches triggered by EXPIRE requests implicitly summarise the actual sequence of state changes by virtue of multiple changes occuring between GET requests.

Keep-alive

Most subscription mechanisms include a keep-alive function. This typically exists for a number of reasons:

  1. To ensure that server-side resources are not consumed long after a client has forgotten about the subscription
  2. To allow a client to determine that the server has forgotten about the subscription and needs a reminder
  3. To allow a client to detect the death of its server

SENA addresses the first point with a long server-driven keep-alive. I have deliberately pointed to a default period consistent with the default required for TCP/IP keep-alive: Two hours. It should be long enough not to significantly increase the base-load of the Internet associated with ping data, while still allowing servers the opportunity to clean up eventually.

The second point is dealt with in SENA by a prohibition against the server-side losing subscriptions. Subscriptions should be persistent across server failures and failovers. Client-side query of subscription state is permitted via a GET to the subscription resource, however this should not be used as a regular keepalive especially over a short period. Essentially, a server should never lose a subscription and user intervention will be required whenever it does happen.

Death detection is not dealt with in SENA. It is an assumption of SENA that the cost of generic death detection outweighs the benefits. The cost of death detection is at least one message pair exchange per detection period. Over the scale of the internet that sort of base load just doesn't compute. Subscriptions should become active again after the server comes online, therefore server downtime is just another time when the basic condition of the subscription holds: That the client has the freshest possible data. Monitoring of the state of the server is left as a specialised service capability wherever it is required.

Benjamin