Sound advice - blog

My recent post on deprecating the POST HTTP method for resource creation generated a few responses: Azubuko Obele, Peter Williams (1) (2), and Stefan Tilkov.

DELETE

First let me clarify for Azubuko Obele that I am not experimenting with deprecating DELETE at the moment. I have some questions about the contrasting concepts of a resource not existing and a resource having the null state, but they can lie for the moment. DELETE is part of HTTP and established REST practice. It has properties that are as good as those of PUT, so there is no real impetus to give it the boot. That part of my discussion was more of a prod to readers: A challenge to justify the existence of every method in use. I have not shaken the idea that DELETE is effectively a PUT of null, but there is really no point in actually practically replacing one with the other at this stage.

Stateless Redirection

On the PUT for resource creation front, Peter explains the problem better than I did originally and has an alternative proposal. The client would PUT to a well-known resource, rather than generating a GUID. That well-known resource would always redirect to a URL of its choice, and the client would follow.

There is a possible flaw in this approach which I didn't cover in my earlier post, and that is the potential for statefulness in the redirection. Consider a server that receives a PUT request to the well-known URL. It says: OK. The next available id is "3", so I'll redirect to there. The next request comes in, and it gets a "4".

It is quite possible that the "3" request will never get through. That means that the server should not prepare any data with that identifier, but should still increment a counter to assign the next id. That's to allow for the later "4" request to come in and be processed properly. It means that any servers within a cluster need to work with a shared counter or counter structure that allows unique identifiers to be assigned despite failover events, parallel processing between servers, and other disruptions. Most databases will have way of making these assignments built in, so it may be possible to use that mechanism. You would just have to make sure that the mechanism would work even when you are not creating actual records, or that any dummy records you create are eventually cleaned up.

What I was thinking originally was that any redirection would be stateless, and would probably have to retain the client-supplied GUID. For example, the server might convert the request URL of <https://example.com/factoryResource?guid=76fd9473-a270-4aac-8a06-e5265048cbbc> to <https://example.com/blogposts/76fd9473-a270-4aac-8a06-e5265048cbbc> almost as a regular expression match and replace, or might use identifying input from the request body to determine the correct request URI. In either case it could then immediately forget about the request. It did not assign an id, so doesn't need to increment any kind of counter. This is a bit ugly. I work in a machine-to-machine architecture where this is possibly less of an issue, but it might be better even in this environment to use readable identifiers. Debugging is an important part of the development process, after all.

Bait and Switch

So what is the alternative? One I perhaps missed describing earlier is to create the resource at the client-specified URL, including GUID, but also make the resource state available at another URL:

PUT /factoryResource?guid=76fd9473-a270-4aac-8a06-e5265048cbbc HTTP/1.1
Host: example.com
Content-Length: ....

<PurchaseOrder>
...
</PurchaseOrder>

The server may respond with:

HTTP/1.1 303 See Other
Location: https://example.com/newResource
...

Unfortunately, See Other isn't a good match. It doesn't carry the semantics we area really after. See Other only suggests that the client GET the nominated URL in order to complete the request. It doesn't indicate what the nominated URL actually means. 2616 demands that if we want to apply a PUT to another resource that 301 MUST be used. However, standards are meant to be bent. We could certainly pull a bait and switch on the client: Let the PUT go through with a simple 201 response, then redirect any later request to the new resource. All of these approaches are a little untidy, though.

Server-provided URL

This brings us back to the question of where the identifier comes from in the first place. If the client is operated by a human, it is likely the URL comes from a server-generated form. In this case no redirection will be necessary. The server should have chosen a good identifier from the get go. It is only when the client has to come up with the id on its own that we get into this redirection tizz. That is the case where the client and server have had no previous contact, except for the user of the client configuring a base URL into the client application.

Coupling Effects

Peter is concerned that I am creating coupling by requiring a particular way of constructing the real URL from the configured base URI in the client. Peter, you slightly misquote my example. I deliberately used the query part of the request URI as the place my GUID goes. My suggestion is that this is how all servers should accept this kind of PUT. When the client is configured with a base URI and has to determine the request URL from the base, that it fill out the query part of the base URI in a standard way. I suggest that the query parts of URIs should be compatible between servers to support non-human REST architectures. I suggest that all servers accepting this kind of PUT use or support the ?guid={GUID} syntax. Coupling, yes, but coupling to a standard approach.

Conclusion

I am continuing along with my experiment for now. Clients start with a configured base URI, and construct a URL by filling out the query part with a GUID. My servers aren't doing any redirection. In the future I may consider making the same object available under another URI without redirecting the original client.

Benjamin

Following recent discussion on rest-discuss, I have been experimenting with deprecating the POST method for architectures my organisation controls, and am eyeing DELETE with some suspicion also. I am suggesting that PUT be used wherever possible.

Pendulums and Muddy Waters

I think there is a pendulum in method definition. Every so often you think you need a few more methods to do special things, but the time comes to swing back and consolidate. GET is a clear winner both in the REST and the non-REST camps, but argument rages over use of the other classic REST verbs. POST is particularly problematic because of its non-idempotent nature. If my POST request times out, I can't just try again. I might be submitting my request twice.

I see value in trying to reduce the set of methods down to GET and PUT. I think it could clear the waters a little and result in simpler architectures. At the same time I acknowledge that asking the question can be seen to be muddying the waters in the short term. I hope I don't trigger too much gnashing of teeth with this suggestion, nor with a suggestion I may make soon about cutting down the HTTP response codes to a "useful" subset. I acknowledge also that I might be pushing some anti-URI-construction buttons with this article. However, I see that also as an issue that needs some clarity and guidelines going forwards. Ultimately, I hope to draft some documents for my company's internal use about the subset of HTTP they should implement and about other aspects of RESTful design. I hope that such documents could eventually be made available to a wider audience, also.

The proposal

My current direction is to combine PUT with client-side "form submission". The classic POST request might be as follows:

POST /factoryResource HTTP/1.1
Host: example.com
Content-Length: ....

<PurchaseOrder>
...
</PurchaseOrder>

The server may respond with:

HTTP/1.1 201 Created
Location: https://example.com/newResource
...

In my experimental approach I require the client to provide a globally unique identifier as part of the request URL. An example request might be:

PUT /factoryResource?guid=76fd9473-a270-4aac-8a06-e5265048cbbc HTTP/1.1
Host: example.com
Content-Length: ....

<PurchaseOrder>
...
</PurchaseOrder>

The server may respond with:

HTTP/1.1 201 Created
...

My architecture is highly focused around machine-to-machine comms, so I expect that most servers will simply create a resource at the location the client requests (including the query part). The actual construction by a client of a request url from the base factory url is anticipated to be standard, which is to say "hard-coded". Every factory resource should become a factory resource family with the same query structure. I am following the direction laid down in one of my recent articles of suggesting that the query component of a url constructed by clients should follow content-type rules of having consistent structure between similar services.

Many services will choose to use their own unique identifiers for the created resources. For example, the guid could be substituted for a monotomically-increasing number or other identfier within a particular context. These services should redirect the PUT request to their preferred url:

HTTP/1.1 301 Moved Permanently
Location: https://example.com/newResource
...

Note that there is still a caveat when attempting to POST/PUT Once Exactly (POE). It is important that created resources are not too short-lived. If they are short-lived the server should keep track of the fact they recently existed in order to inform the client that is creation attempt was successful by way of a 410 Gone response to subsequent PUTs. A similar problem arises if the state of the resource is updated soon after resource creation. The client who thinks the creation request failed could blat over the changed state. Perhaps there still does need to be some separation between a method that creates state at a particular url and one that replaces state. Perhaps there does need to be separation between a CREATE method and an UPDATE method. Too late now for HTTP on the Web, perhaps.

Conclusion

Pub/Sub aside, GET is the right way to transfer state from server to client. However, the waters are muddier when transferring from client to server. I suggest that special applications of PUT may be more useful than defining separate methods. PUT, POST, and DELETE currently mean "create or replace state where I tell you", "create state wherever you like", and "eliminate state" or "replace with the null state". My gut suggests that collapsing all of these methods into PUT is a net win. My only hesitation is in the misuse PUT might see if the "deprecate POST" message gets out too far. Therefore, I suggest PUT be used where state is being transferred idempotently. POST should still be used where tunnelling or "process this" semantics are required. POST should be a warning for any code that sees it to tread carefully: The guard ropes have been cut.

Benjamin

I have postulated before about REST's uniform interface consisting of a triangle of identifier type, method or interaction type, and content type. Perhaps it would clarify matters for the WS-* camp if another dimension were added.

Semantic Confidence

At WSEC there were a number of people in the room worried about the lack of semantics of POST. An old question kept coming up: Could a client accidentally trigger a harmful operation by accident? Perhaps the launch of a nuclear weapon?

Security issues aside, the question can be posed as: "How can the client be sure that its request is understood?", or "How can the client be sure that the server will do what it wants the server to do?". The question is posed around POST, but the same can be asked for PUT or even safe methods such as GET.

A PUT of text/plain "1" could save the result of a calculation into a loan amortisation tool, or could change the US defence condition and set in motion countermeasures that launch the world's nuclear missiles. However, suggesting that the client should have coded knowledge of which case will occur is missing the point. There are many cases of URLs between the benign and the disastrous for which it is useful to configure a client to send its text/plain "1". Instead of saving data into a loan amortisation tool, perhaps the data can be stored in an excel spreadsheet. Perhaps a rival's tool will one day be preferred. Just how much of the server-side semantics should be encoded into the message?

Just as the same PUT request could mean different things when directed to different resources, GET means different things to different resources. If I GET https://google.com/ I expect a form to do web searching. If I GET https://en.wikipedia.org/ I expect a navigable encyclopedia. In the pure machine-to-machine world I might point a calculations engine at a resource demarcating the state of a circuit breaker, or a water pump, or a fan. Depending on the calculation these may all be valid configurations.

Interoperability

REST is about interoperability, and at its heart we see the answer to our puzzle of where the resource-specific semantics "go" in a message. They don't go. They are held implicitly in the URL, a URL that fundamentally a human has provided to the client. Whether the URL was provided by explicit configuration, by hyperlinking from other documents, or any other source... there is a reason this client is making this request to that resource at this time. A human has made all of the necessary logical connections. It is the machine's job to get out of the way of this human decision.

So taking GET as the guide we can draw the same implications for PUT, DELETE, and even our scruffy old friend POST. It is up to the resource to determine how to interpret the request, so long is it does so within reason. We don't introduce unnecessary coupling just so the client feels like it has more control over the situation. Feed as many semantics down the pipe as you like. The server still decides how to interpret your request. All you are doing by encoding this information directly is preventing interoperation with other resources.

Human-directed Interconnection

So the REST triangle stands, firm and proud. One set of schemes for identifiers in the architecture, one set of interactions that components in an architecture participate in, and one set of content types exchanged within an architecture. Every message is a permutation of these separately-evolving facets of the message, and can be correctly handled based on these facets. Additional semantics never leave the server, however they are identified by a URL. Each URL is transferred from server to client in preparation for any request under human direction, and is transferred back to the server as the request URL. Humans are free to connect components together when it makes sense to do so, and coupling at the technology level does not get in the way of that human direction.

The only thing that should get in the way of free human connection of components is when the interaction is not meaningful for a particular resource, or the schema of data transmitted and required between components does not match. In these cases it is possible to introduce configuration errors by mismatching client and server. However, this is the least of your worries when you are configuring components to talk to each other. Again, we have the amortisation calculator vs defence condition case. A human must have enough information at hand to avoid confusing the two. The core problem remains knowing which resources are out there, and conveying that information to the human along with human- or machine- readable semantics as to what will occur when a request is sent to a particular url.

While it may be useful to introduce a family of languages that helps guide a human through the configuration process, only the final selected url should actually be transmitted within the operational architecture. "getStockQuote" should never appear in the message, nor "getUnitPrice", nor "getCurrencyAmount". Simply using "GET" with the appropriate URL is the right way to ensure that as many clients as possible who want to access the available information can access it.

Conclusion

It is easy to fall into the trap of thinking that a more explicit client request is more valuable or more certain than a uniform request. However, while this approach necessarily introduces negative coupling it does not introduce a strong balancing positive effect in terms of semantic confidence. All of that confidence comes from an out-of-band trust relationship with the URL a request is directed to, not from the message itself. If a message that uses uniform identifiers, uniform interactions and uniform content types can convey the same information it will always be more valuable to go down the uniform path. Where no uniform message can convey the information you want, REST still has value. It will still be more valuable and cost-effective to change a single REST triangle facet than to reinvent the wheel and start from scratch.

Benjamin

URI-construction is usually seen as a bad thing in REST practice. However, just about everyone does it in some form or another. You start with some sort of base URL, and you fill out variable parts of it to construct a URL that you actually look up or submit data to. We refer to experience on the Web and say that it is alright to construct a URL sometimes. In particular, it is OK to construct a URL when you have a document that you just obtained from the server that tells you how to do it. This kind of URL construction isn't really a problem. In fact, it is just an advanced form of hyperlinking. So what are the limits of this form of hyperlinking when there is no human in the loop?

Form Population for Machines

The distinction between a human-submitted form and a machine-submitted form is an important one. A form intended for a human will include textboxes and other user-interface widgets alongside supporting text or diagrams that explain how to fill the form out. A machine cannot understand the kinds of loose instruction given to a human, so we have to consider how a machine knows what to put where in the URL.

I think that the simple answer is that the input to any machine-populated form is effectively a standard document type, or at least a standard infoset. The machine that constructs a URL does so from a particular set of information, and the form acts as a transform from its infoset into the output URL.

For example, a machine that is constructing a query for the google search engine must know to supply a google-compatible search string in the "q" field of the form. A client of yahoo must currently know to supply a yahoo-compatible search string in the "p" field. While humans are able to fill in forms that accommodate these differences, machines are more limited. If we are ever to have any hope of machine-submitted forms we will need to look at reducing this kind of deviation in infoset structure.

Transformations from Infoset to URL

This characterisation of a machine-populated form as a transform should impact how we go about selecting and evaluating forms technologies that are destined purely for machines to populate. In particular, XSLT jumps right up the list in terms of possible technologies. It is already a well-established transform technology that can output text as well as other structured document types. If we view the source document as either XML or an XML infoset, XSLT provides an obvious way to achieve the URL construction goal.

Another approach would be to look carefully again at how Web forms work. In practice, they construct an infoset based on user input then use a defined mechanism to transform the infoset into into the query part of a URL. This could be a significantly simpler approach than even embedding an XSL engine. Let's say we see the source document as an XML infoset again, we can follow the rules that XForms defines for transforming an XML document into a query. These rules are essentially that elements with text nodes are turned into URL parameters.

Coupling Effects

On first blush this standard transform approach looks like it couples client and server together, or requires different server implementations to obey the same rules for their URL-space... and that is not entirely false. The factors that limit these effects are the use of a standard document type as input to the transformation, and the ability for server-side implementations to redirect.

In the client-to-server coupling case, we often see service-specific URL construction occurring in clients today. Instead, this construction should be able to be applied to different services that have similar construction requirements. I should be able to start up a search engine rival to google and use the same infoset as input to my URL construction. The client code should accept a base URL as input alongside other parameters that form the infoset, meaning that all clients need to do to use my rival service is change their base URL. Code changes should not be required.

In the server-to-server coupling case, this is an interesting problem. We usually see content types and methods needing to be standard, but give freedom to servers to construct their URI-spaces in whatever way they see fit. The XSLT form submission method would give them that freedom up-front, however redirection is also a way of achieving that freedom. A simple 301 Moved Permanently would allow the server freedom in how they construct their URLs. Greater freedom, in fact, than XSLT running in a client implementation could because it would have more information at its fingertips with which to decide on the redirection URL. To achieve this, all we really need to sacrifice on the server side is a single base URL with a query space that matches a standard infoset that machine clients can be expected to have at hand ready for submission.

Conclusion

My considered view is that using the query part of a URL as a way to pass a standard infoset to a server is a valid way of constructing URLs. I think it is the simplest and most robust way to transform an infoset into a URL, and possibly the most powerful. Current attempts to allow the server greater freedom as to how it constructs its URLs are interesting, but at this point I do not intend to implement anything but the query-part-as-content approach in my development. I think the focus should shift away from this technical sphere of URL construction to a process of defining the content types that are fed into these transforms.

Benjamin

REST is often strongly correlated with HTTP and its verbs. It is often contrasted with a SOAP or WS-* services as two opposing technologies or approaches. I take more of a middle ground. I think that you are already doing REST. The fundamental questions in the development of your network architecture are not necessarily whether or not you should be doing REST, but specifically what benefits do you intend to extract from the development. Let me run you through how I see this thing playing out.

Your messages are already uniform

Working from first principles with the constraints, REST says that you should exchange a constrained set of message types using standard methods and content types. So you have your IDL file or your WSDL, and you have a number of methods listed in that file. If you are using the document-oriented style of SOA your WSDL will no doubt include or refer to the definition of a set of documents. In other words, your WSDL defines the scope of your web. Everything in that web... everything that implements the WSDL either as a client or as a server... can meaningfully exchange messages. These components of your architecture can be configured together. They don't need to be coded. A human can decide to plug one to the other arbitrarily without the technology getting in the way.

But the technology is getting in the way.

Your uniform methods aren't achieving network effects

You have defined this web, this WSDL, this architecture... but it is too specific. You can only connect the two components together that you designed the interface for, or you can only connect the client apps to the server that you designed the interface for. It isn't a general mechanism for letting a client and server talk to each other, because the problems of that particular interaction are built into your web in a fundamental way that makes solving other problems difficult.

That's ok, isn't it? If I want to solve other problems I can create another WSDL. I can create another web. Right?

You can, and sometimes that is the right approach. However you impose a cost whenever you do that. You can only plug components together if they are on the same web. You can only plug them together if they share a WSDL. Otherwise you have to code them together. Most of us have been writing code whenever we want two network components to talk to each other for so long that we assume there is no alternative. However, I come from the SCADA world and an increasing number of competent people come from the Web world. Experience in both of these worlds suggests we can do better. But how much better, exactly?

In an ideal world...

The ultimate ideal would be that so long as two machines have the same basic data schema and any particular interaction makes sense, that they can be configured to engage in that interaction rather that requiring us to write special code to make that interaction happen. However, is this practical? What is achievable in practice?

The Web sets the benchmark by defining separately the set of interactions machines participate in and the et of document types they can exchange. The three components of what make up our messaging change and evolve at different rates, so separating them is an important part of solving each of these important problems.

1. How we identify participants in an interaction, especially
   • request and response targets
2. What interactions are required, including
   • Request Methods
   • Response Codes
   • Headers
   • Transport Protocol
   • TCP/IP Connection direction
3. How information is represented in these interactions, including
   • Semantics
   • Vocabulary
   • Document structure
   • Encoding (eg XML), or Serialisation (eg RDF/XML)

Whether or not you can actually achieve consensus on all of these points is a difficult question, and usually limited by non-technical issues. You really need to hit an economic sweet spot to achieve wide-scale consensus on any part of the trio. Luckily, consensus on identification and interactions is widely achieved for general-purpose problem sets. Special uses may need special technology, but URLs and HTTP go a very long way to closing out this triangle of message definition. The remaining facet is perhaps the hardest because it requires that we enumerate all of the different kinds of information that machines might need to send to each other and have everyone in the world agree to that enumeration.

So this is the limiting factor of the development of the Semantic Web, a web in which rich semantics are conveyed in messages that are understood by arbitrary components without having to write special code. The limiting factor is the number of kinds of information you can achieve global consensus on. However, we don't really need to have global consensus for our special problems. We only need consensus within our little web. We just need to understand the messages being exchanged in a particular enterprise, or a particular department, or a particular piece of IT infrastructure. We just need the components of our web to understand.

Closing the triangle: Content Types

So if the web relies on special understanding of this final facet, what is the point of agreeing on the first two? The answer to that, my friends, is evolution. What is special today might not be tomorrow. I can develop a format like atom to solve a specific set of problems within my web, and then promote my format to a wider audience. The more components that implement the document type, the wider my web and the bigger the network effects. The other two facets already have widespread consensus, so I can target my efforts. I can avoid reinventing solutions to how objects are named or how we interact with them. I can just focus on the format of data exchanged in those GET, PUT, and POST requests. The rest is already understood and known to work.

Now that's all well and good. The Semantic Web will evolve through solutions to specific problems being promoted until individual webs that solve these problems are joined by thousands of components operated by thousands of agencies. But... what about me? What about today's problems? Most of my document types will never leave the corporate firewall, so is there still and advantage in considering the Web's decomposition of message type?

I suggest, "yes". Whenever you integrate a new component into your network, do you need to write code? When new protocols are defined, are the easy to come to consensus on? As an integrator of components myself I find it useful to be able to fall back on the facets of message type that are widely agreed upon when new protocols are being defined. We don't have to go over all of that old ground. You and I both know what we mean when we say "HTTP GET". Now we just have to agree on specific urls and on content types. Chances are that I have a content type in my back pocket from similar previous integrations or that I can adapt something that is in use on the wider Web. Any message exchanges that could use pure Web messaging does so, and anything that needs special treatment gets as little special treatment as possible.

Certainly, after a few years of doing this kind of work it gets easier to build bridges between software components.

Vendors and Standards

Unfortunately, this sort of gradual evolution and interactions between the wider webs and your special case are not well supported by WS-*. Hrrm... this is where I find it hard to continue the essay. I really don't know WS-* well enough to make definitive statements about this. What I can do with HTTP is easily add new methods within the namespace of the original methods. I can then promote my new methods for wider use, so for example I can promote a subscription mechanism. In XML I could add extensions to atom, and if I used the atom namespace for my extensions they could eventually be adopted into atom proper without breaking legacy implementations of the extensions. Can the same be said for WS-*? Does it allow me to separate the definitions of my methods and my document types effectively for separate versioning? Do the tools support or fight these uses of WS-*?

For that matter, do the tools associated with XML encourage must-ignore semantics that allow for gradual evolution? Do they encourage the use of a single namespace of extensions, with alternative namespaces only used for separately-versioned sub-document types such as xhtml within atom? My tools do, but they are all written with the architectural properties I require in mind. Does the world and do vendors really understand this communication problem? Do they understand the importance of limiting the number of kinds of messages that exist in the world? Are they taking practical steps to make it easier to reuse existing messages and message structure than to create incompatible infrastructure? Do programmers understand the social responsbility that defining new message types places on them?

Simplicity: Architecture or toolchains?

REST is sometimes bandied about as a simpler approach than WS-*, and certainly REST architectures are simpler. They have less variation in message types, and promote configuration of components rather than coding to make new interactions work. However REST only achieves this by shifting the social problem out of the software. Instead of solving the problem of how two components interact with one-off software, we have a long multi-party standardisation effort that ensures thousands can interact in this way. REST encourages the reuse of existing messages and structure, but in truth it is often easier to add a new message type simply by throwing an extra method into your WSDL. REST results in less code and simpler architectures. SOA results in more code and more complex architectures... but the difference isn't between HTTP and SOAP. It is between investment in an architecture and investment in a toolchain.

Perhaps that is the take-home, ladies and gentlemen: You can achieve better and simpler and higher-value architectures... but even when you leverage standard identification schemes and interaction models there is no silver bullet. You still need to choose or evolve or invent your document types wisely. That costs time and money, as does promoting anything you do invent to sufficient scale to achieve significant value. That effort has to be judged against the value of the network to you and your company. On the other hand, I think we are lacking the tools that make some of these problem easy to identify. I think we can make it easier to build new infrastructure based on the consensus we have already achieved. I think we can do better.

Conclusion

You are already doing REST, but are you getting the network effects that you see on the wider Web? Can a million different software components carry on perfectly sensible conversations with others that they have never met before and had no special code written for? Can you do REST better? Is it worth the extra effort, and what tooling do we need to put the evolvable Semantic Web within the reach of mere mortals?

Benjamin