Sound advice - blog

Tales from the homeworld

My current feeds

Sat, 2007-Jul-14

The War between REST and WS-*

David Chappell posits that the "war" between REST and WS-* is over. The evidence for this is that platforms such as .NET and Java that have traditionally had strong support for WS-* are now providing better tools for working with REST architectures.

The War

Is the war over? I think it we are getting towards the end of the first battle. WS-* proponents who see no value in REST are now a minority, even if some of those who accept REST's value are only beginning to understand it (both REST, and its value). Moreover, I think that this will in the long run be seen as a battle between Object-Orientation and REST. That battle will be fought along similar lines to the battle between Structured Programming and Object-Orientation.

In the end, both Object-Orientation and Structured Programming were right for their particular use case. Object-Orientation came to encapsulate structured programming, allowing bigger and better programs to be written. We still see structured programming in the methods of our objects. The loops and conditionals are still there. However, objects carve up the state of an application into manageable and well-controlled parts.

My view is that the REST vs Object-Orientation battle will end in the same way. I believe that REST architecture will be the successful style in large information systems consisting of separately-upgradable parts. I take the existing Web as evidence of this. It is already the way large-scale software architecture works. REST accommodates the fact that different parts of this architecture are controlled by different people and agencies. It deals with very old software and very new software exist in the same architecture. It codifies a combination of human and technical factors that make large-scale machine cooperation possible.

The place for Object-Orientation

We will still use Object-Orientation at the small scale, specifically to build components of REST architecture. Behind the REST facade we will build up components that can be upgraded as a consistent whole. Object-Orientation has been an excellent tool for building up complex applications when a whole-application upgrade is possible. Like traditional relational database technology, it is a near perfect solution where the problem domain can be mapped out as a well-understood whole.

Hard-edged Object-Orientation with its crinkly domain-specific methods finds it hard to work between domains, and nearly impossible to work across heterogeneous environments where interacting with unexpected versions of unexpected applications is the norm. Like Structured Programming before it, the Object-Orientated abstraction can only take us so far. An architectural style like REST is required to build much larger systems with better information hiding.

Conclusion

To me, the "war" is over. REST wins on the big scale. Object-Orientation and RDBMS win at the small scale. The remaining battlefield is the area between these extremes. Do we create distributed technology based on Object-Oriented principles using technology like Corba or WS-*, or do we construct it along REST lines?

Like David, I see the case for both. Small well-controlled systems may benefit from taking the Object-Oriented abstraction across language, thread, process, and host boundaries. However, I see the ultimate value of this approach as limited. I think the reasons for moving to a distributed environment often relate to the benefits which REST delivers at this scale, but Object-Orientation does not.

Addendum

Mark Baker picks up on a specific point in David's article. David says that REST is fine for CRUD-like applications. Mark essentially counters with "incorporate POST into your repertoire". This is where I have to disagree. I think that any operation that is sensible to do across a network can be mapped to PUT, DELETE, GET, or SUBSCRIBE on an appropriate resource. I see the argument that POST can be used to do interesting things as flawed. It is an unsafe method with undesirable characteristics for communication over an unreliable network.

My CRUD mapping is:

C or U
PUT
R
GET
D
DELETE

I then add trigger support as SUBSCRIBE.

Mark's example is the placement of a purchase order. I would frame this request as PUT http://example.com/purchase-order/1000 my-purchase-order. This request is idempotent. My order will be placed once, no matter how many times I execute it. All that is needed is for the server to tell me which URL to use before I make my request, or for us to share a convention on how to select new URLs for particular kinds of resources. Using POST for this kind of thing also has the unfortunate effect of creating multiple ways to say the same thing, something that should be avoided in any architecture based on agreement between many individuals and groups.

In my view, the main problem with the CRUD analogy is that it implies some kind of dumb server. REST isn't like this. While your PUT interaction may do something as simple as create a file, it will be much more common for it to initiate a business process. This doesn't require a huge horizontal shift for the average developer. They end up with the same number of url+method combinations as they would have if they implemented a non-standard interface. All they have to do is replace their "do this" functions with "make your state this" PUT requests to an appropriate url. REST doesn't change what happens behind the scenes in your RDBMS or Object-Orientated implementation domain.

Benjamin

Sun, 2007-Jul-08

HTTP Response Codes

There are too many HTTP response codes.

One of the first questions asked when someone tries to develop an enterprise system around is which of these codes is important to support in client applications. The short answer is "all of them", but it comes down to which codes mean something to your application. The following table summarises my defaults for automated clients. Drop me a line if you have comments or suggestions, <benjamincarlyle at soundadvice.id.au>. Codes are from rfc2616, HTTP/1.1:

CodeReason PhraseTreatment
100 Continue Indeterminate
101 Switching Protocols Indeterminate
1xx Any other 1xx code Failed
200 OK Success
201 Created Success
202 Accepted Success
203 Non-Authoritative Information Success
204 No Content Success
205 Reset Content Success
206 Partial Content Success
2xx Any other 2xx code Success
300 Multiple Choices Failed
301 Moved Permanently Indeterminate
302 Found Indeterminate
303 See Other Indeterminate
304 Not Modified Success
305 Use Proxy Indeterminate
307 Temporary Redirect Indeterminate
3xx Any other 3xx code Failed
400 Bad Request Failed
401 Unauthorized Indeterminate
402 Payment Required Failed
403 Forbidden Failed
404 Not Found Success if request was DELETE, else Failed
405 Method Not Allowed Failed
406 Not Acceptable Failed
407 Proxy Authentication Required Indeterminate
408 Request Timeout Failed
409 Conflict Failed
410 Gone Success if request was DELETE, else Failed
411 Length Required Failed
412 Precondition Failed Failed
413 Request Entity Too Large Failed
414 Request-URI Too Long Failed
415 Unsupported Media Type Failed
416 Requested Range Not Satisfiable Failed
417 Expectation Failed Failed
4xx Any other 4xx code Failed
500 Internal Server Error Failed
501 Not Implemented Failed
502 Bad Gateway Failed
503 Service Unavailable Repeat
504 Gateway Timeout Repeat if request is idempotent, else Failed
505 HTTP Version Not Supported Failed
5xx Any other 5xx code Failed

Key:

Repeat
The client SHOULD repeat this request, taking into account any delay specified in the response. If the client chooses not to repeat the request, it MUST treat the transaction as a failed.
Success
Do not repeat. The client SHOULD treat this as a successful transaction, however specific codes may require more subtle interpretation in unusual environments.
Failed
Do not repeat unchanged. A new request that takes the response into account MAY be generated. The client SHOULD treat this as a failed transaction if a new request cannot be generated, however specific codes may require more subtle interpretation in unusual environments.
Indeterminate
Do not repeat unchanged, but this code MUST be understood. A new request that takes the response into account SHOULD be generated. If the client is unable to generate a new request this code MUST be treated as Failed.

Redirection

Redirection is essential to the identifier evolution mechanism in the uniform interface. Consider a typical client: It consists of configuration and code that enables it to interact with resources elsewhere in the network. However, the configuration may become out of date if it is installed in a client that runs over a very long period of time. The url spaces the client is configured to interact with may be restructured. Redirection response codes allow the servers that provide a particular url space to reconfigure their clients and allow them to keep operating.

Note that the HTTP specification requires reasonable limits to be placed on redirection, and demands human confirmation in cases where redirection might cause the request to mean something different to what the client intended. The right way to deal with this is usually to allow the user to define limits to the way redirection is handled. For example, a redirection profile may be supplied that only allows redirection to urls that have the same domain name as the original url. Redirection loops also need to be explicitly defeated.

Some commentary:

CodeReason PhraseCommentary
100 Continue I'm not a fan of this code. It adds latency to valid transactions, optimising for the uncommon failure case. It adds a "multi-callback" mechanism. It returns multiple responses for a single request. I have put this kind of mechanism into place in proprietary systems I have developed. Unfortunately, this multi-callback mechanism isn't implemented in any kind of general way. My experience from building systems with multi-callbacks also suggests that they don't really work in fast-failover high availability environments without an excessively-expensive keep-alive mechanism.
101 Switching Protocols An upgrade path to WAKA or HTTP/2.0, if it ever arrives. Personally, I think we'll still have devices speaking HTTP/1.x (possibly still HTTP/1.0) in a fifty to a hundred years. Any protocol that hopes to displace HTTP/1.x has to do so on the Web to be successful. That's a huge undertaking.
200 OK Yay, success
201 Created Equivalent to 200 OK for a PUT. When returned from POST includes a useful Location field. However, POST is non-idempotent and unsafe. My preference is not to use it, especially in pure machine-to-machine architectures.
202 Accepted It is hard to know what to do with this code. It is equivalent to an SMTP-style store-and-forward. The best that an automated client can do is assume success. Perhaps a way to retrieve the real response asynchronously will be devised some day.
203 Non-Authoritative Information Only useful for GETs, and returned by a proxy.
204 No Content This is really a terrible code. It tells a HTTP response parser not to expect a body, meaning that there is really no other way to say this. It would be nice to separate out the content/no-content decision from the response code. It has the side-effect (or main purpose, depending on your perspective) of telling a web browser not to replace the current page.
205 Reset Content This is another brower-directed code. It doesn't really make sense for automated clients, except at yet-another-success code.
206 Partial Content Success for a partial GET, an artifact of not knowing in advance whether the server supports such a thing. This sort of thing reveals the evolution that HTTP has gone through over the years, and is a bit creaky. However, that evolution also demonstrates the essential method evolution mechanisms available in HTTP.
300 Multiple Choices Content negotiation by different URLs. Possibly not the best approach, but content negotiation is essential for an architecture that has an evolving content-type pool. 300 Multiple Choices, or the Accept header? So far Accept seems to be out in front. The main weakness of this code is probably the lack of a defined automatic negotiation mechanism. Automated clients can't really make use of this code.
301 Moved Permanently Essential identifier evolution code
302 Found More identifier tweaking
303 See Other A request in two parts: A second request is required to retrieve the result of the first. Is this a better "202 Accepted"? Perhaps the second request can return 503 Service Unavailable until the response is ready.
304 Not Modified This response code is specific to GET, and indicates that the client already has an up-to-date representation of the resource state. It isn't relevant to other response codes, and is essentially another success code.
305 Use Proxy This is a potentially neat way to introduce an intermediary. Do you need to introduce a specialised authenticating proxy? Perhaps you want to introduce a proxy to validate schemas, and otherwise police service-level agreements. If clients understand this code, all of this is possible.
307 Temporary Redirect Another useful code. Not sure you are ready to commit to a permanent move? Do you need to set up a temporary service during transitional arrangements? Ensuring clients support this code allows freedom in how services are managed during these times.
400 Bad Request This is where things get boring for automated clients. A typical automated client knows what request it wants to make to which URL. If the server is unable to process the request for any but the simplest of reasons, there is nothing a typical client can do. Log the failure, and do your best to continue working. Only a human can really fix many of these problems, so distinguishing between them in a machine-readable way doesn't carry as much value as you might think. In those cases the reason phrase or enclosed entity will be of more use than the code itself.
401 Unauthorized This code is used when an automated client doesn't want to supply credentials by default. It gets the code, and retries with credentials. If that doesn't work the client can try any alternate credentials it might have, but this set is typically limited.
402 Payment Required Don't use, doesn't work. Maybe one day the whole Web will work like the Web we see on mobile phone networks, where we have a pay-per-view controlled content world. I'm sure there are interests out there who want this. From an economic perspective it seems likely that there will eventually be some level of consolidation and reorganisation of the supply relationship on the Web. However, for now different channels are in use when payment is required.
403 Forbidden "Please don't call again". An automated client can't do anything with this code, except give up.
404 Not Found Give up, client... unless you were trying to DELETE. If you were, you just might have succeeded.
405 Method Not Allowed Please try a different method... for "GET"? If the client want to participate in a GET interaction, another interaction won't do as a substitute. This method may allow for evolution in the method-space of HTTP, so probably isn't a complete write-off. However, the set of methods in HTTP is fairly stable. It needs to be.
406 Not Acceptable I can't interact with you. Give up.
407 Proxy Authentication Required Similar to 401 Unauthorised: useful.
408 Request Timeout This method looks like 503 Service Unavailable, but indicates that the client is at fault. Should this be a "Repeat" or "Failed" response? It's hard to tell. The spec certainly allows for a repeat, but if the client continues to fail to deliver its request on time the response will continue to be returned. I'm 50/50 about this. If you can't get your request through once, perhaps the chances of ever getting it right seem less than certain.
409 Conflict "You can't send this request now, but maybe if something changes you can try again. It's not you, it's me. Can we still be friends?". Don't put up with it. Dump that server. Log it, and move on.
410 Gone Your configuration is out of date. What's a client to do, but give up? Again, log it and leave it.
411 Length Required "You aren't giving me enough information to proceed". Well? That's how I was written. Nothing you can do.
412 Precondition Failed "You asked me to do this only if that was true. Well, it wasn't.". Is it a success code because the request was fully fulfilled, or a failure? This code can be used to resolve conflicts in environment, but your client has to know how to resolve conflicts. That typically involves consulting a user. If you didn't ask for this code you won't get it. If you did ask for it, you'll know what to do with it.
413 Request Entity Too Large "Ha, stopped at the firewall". Log it and give up.
414 Request-URI Too Long Ditto
415 Unsupported Media Type Is this a code for content negotiation of a PUT request, or simply to refuse to accept a particular Content-Encoding header. The lack of a list of acceptable content types suggests that is only the latter. You might be able to resubmit in a way the server might accept, but your chances are slim. You and the server don't seem to be on the same page, and submitting request after request to figure out how to communicate isn't necessarily going to help.
416 Requested Range Not Satisfiable Specific to Range request header. If you didn't ask for it, you won't get it. You need to forget about partial GETs for the moment and figure out where you are at again with the server.
417 Expectation Failed The client requested a particular extension or set of extensions to the protocol that the server doesn't support. What do you do? Why did you need the extension? Are these kinds of extension valuable enough to offset the complexity involved in dealing with older servers? Besides: Must-understand is so passé.
500 Internal Server Error HTTP tries to be nice in separating out things that are the server's fault and things that are the client's fault. Unfortunately, we have already seen cases where the classification is ambiguous. Worse, whether the client or server is at fault isn't all that useful a classification for automated clients. It is more important to know whether it can retry the request safely, or not. Internal Server Error is no good for this. It will often have come from back-stop exception handling code in the web server, or something equally generic. Something might have happened. The whole operation might have been wholely successful, or not at all. The only thing we can figure out as a client is that reissuing the request isn't a solution. If we are extremely lucky it might actually help, but trying is just going to mask a problem and increase network traffic. This is one to give up on.
501 Not Implemented How this differs from 405 Method Not Allowed is a bit of a fine point, and the two responses should be treated the same way. This one indicates that no resource provided by this server supports the method. In the modern days of servlets this is a hard statement to make. It seems like 405 Method Not Allowed is the right answer when a method is not implemented by a particular resource. Unfortunately, 405 requires that we also list the methods that are valid to use... not that the client is likely to be able to make use of this list. Generating this list and keeping it in sync with reality is harder that you might assume in various web frameworks: Rock, hard place.
502 Bad Gateway The proxy or gateway you are talking to thinks that the next server up the line doesn't speak valid HTTP. The request was sent, but did it work? Unknown. We know that the same thing is likely to happen if we resubmit the request, so we need to give up.
503 Service Unavailable The service is temporarily down, and may provide instruction about when to retry. This can be a useful code that all clients should support.
504 Gateway Timeout This code is very similar to 502 Bad Gateway. In the 502 case our proxy didn't understand the response from the upstream server. In 504, it didn't receive the response. Either way, the request was sent. Either way, we don't know what happened. However 504 models a reasonably common case of a server failing during a request. For this reason it is useful to generate from libraries that speak HTTP on behalf of any server that times out. Retrying makes sense in this environment, though care should be taken to avoid retrying forever. After all, it may be a real timeout. Perhaps the server simply can't process your request and return a response in the time being allocated to it. This highlights a weakness in HTTP whereby the behaviour of a failed server can be aliased with the behaviour of a slow server. A better solution to detecting server death would be to send keep-alive messages down the connection at regular intervals ahead of the pending response. The exact solution could be a client-initiated ping while requests are outstanding, and pong messages are returned before the response. Alternatively, a standard-mandated but configurable HELO message that is sent at a regular interval would do. TCP keepalive in one or more directions may be a way to achieve this without fundamental protocol changes. A server that is processing a long request but can still maintain ping traffic won't be confused with a dead server. If the server is stuck in some kind of tight loop but continues to manage keepalives it is up to something on the server side to eventually kill it off and fix the problem for the client. This is what I call a "High Availability profile for HTTP", supporting individual clients with fast failover requirements.
505 HTTP Version Not Supported You no speak-a the HTTP? Give up.

Benjamin

Sun, 2007-Jul-01

Nouns, Verbs, Uniform Interfaces, and Mature Architecture

The verbs in a typical RESTful system are fairly simple and easy to understand. PUT, GET, and DELETE are conceptually straightforward. A client sets the state of a named resource. A client retrieves the state of a named resource. A client destroys a named resource. However, the devil is in the detail.

The Linguistic Perspective, or the Uniform Interface?

It is tempting to view REST from a linguistic perspective. We can say things like "we are trading more nouns for fewer verbs", and take the Web as an example as to why this is a good thing. However, it is difficult from a purely linguistic perspective to see exactly where the goodness comes from. Why is it better to have more nouns and less verbs? Where does the uniform interface fit into this? Do I have a uniform interface when I use uniform verbs, or is something more required?

I like to turn this analysis around. Instead of starting with the linguistic perspective, I start from the uniform interface perspective. Let me define the uniform interface:

We have a uniform interface when every component within an architecture can arbitrarily and successfully engage in an authorized interaction with another component in the architecture without prior planning.

The Uniform Interface and REST

In REST we divide the set of components in our architecture into clients and servers. Clients are anonymous and unaddressable in the general case. Servers contain resources that are addressable. When I say every component can talk to every other component we already have to take this with a pinch of salt. In REST we can talk about every client being able to interact with every resource.

We can add another pinch of salt: It isn't always meaningful or sensible for a particular client to talk to a particular server. They might move in different machine "social circles", and talk about different kinds of things. They may develop their own jargon and sub-culture. In fact, machines tend to follow the needs and inclinations of their human users. The fact that machines tend to cluster in sub-architectures shouldn't come as any kind of surprise. It isn't always going to be meaningful for a Web-enabled coffee machine to talk directly to a nuclear power plant's monitoring system. It isn't always going to make sense for a personal video recorder who is only interested in program listings to talk to a weather service.

This means that we can only really talk about the uniform interface in terms of any client of a sub-architecture being able to engage in authorized interactions with any resource in the same sub-architecture. Sub-architectures overlap in the same way that sub-cultures do, so this statement about the nature of uniform interfaces is a fairly fluid one. Value is achieved by participating in the largest sub-architectures, and by keeping down the overall number of sub-architectures your software participates in.

The Uniform Interface and Protocol

Now that we are looking from the uniform interface perspective, it pays to take another look at our verbs. It soon becomes apparent that "GET" is not enough. We need our client and server to agree on a lot more than that verb in order to interact. Client and server must agree on a protocol. The whole of the message fashioned by the client needs to fits with the server's mechanisms for interpreting that message. More-over the response that the server generates is part of the interaction. The client must be able to process the message that the server returned to it.

With the uniform interface view firmly in our minds, we can look back over the mechanisms that REST uses to decouple parts of its protocols. In a sense, it might not matter that we decouple identification from verbs and from content type. If one protocol governs a particular sub-architecture, it doesn't matter how it is composed. It only matters that the components of that sub-architecture can speak the protocol.

However, taking this perspective misses one of the key features of the REST style: It is designed to allow our protocol to evolve. Experience suggests that controlling the specification of the URI, the transport protocol (including verbs, headers, and response codes), and content types provides excellent characteristics for evolution.

Participating in sub-architecture

REST is the style for a new kind of architecture. In days of yore we might have expected that we would replace the architecture of our old system with each new system we deploy. We could afford in those days to rip out systems wholesale, and replace them with new systems. I suggest that this is no longer the case. It is increasingly important to be able to replace systems within an architecture, rather than replacing the architecture per se. Other systems need to be able to deal with evolution within their architecture. Protocols and architecture need to be well beyond SOA-class. They need to be Web-class, and Web-style.

Looking back from the uniform perspective lets us see the linguistic debate in a new light. Of course it is possible to introduce new verbs If we are willing to break the uniform interface. Of course this allows us to map the interactions we have within our system onto fewer resources or network-exposed objects. However, doing so greatly harms our opportunities for network effects. Our sub-architecture is reduced to the set of components that directly speak this new protocol we define, and there is reduced opportunity for servers and clients to participate in overlapping sub-architectures. The exact balance between nouns and verbs is a matter for ongoing community debate, but using custom verbs where standard ones would do is an essentially indefensible approach. Just by introducing the necessary resources is possible to prevent divergence from the uniform interface. It is possible to increase the value, evolvability, and overall maturity of your solution. You can always map the methods of these resources onto a smaller set of objects behind the protocol curtain.

Conclusion

The balance between nouns and verbs is a matter for the community that governs a particular architecture. Going it alone is counter-productive, potentially risky, and usually unnecessary. Instead of looking to custom verbs, think more about your content types. That is where most of the real challenges of protocol evolution and maturity lie.

Benjamin