Sound advice - blog

Thanks to Robert Collins for your input on my previous blog entry about using the method name in a HTTP request as effectively a function call name. Robert, I would have contacted you directly to have a chat before publically answering your comments, but I'm afraid I wasn't able to rustle up an email address I was confident you were currenly prepared to accept mail on. Robert is right that using arbitrary methods won't help you get to where you want to go on the Internet. Expecting clients to learn more than a vocublary of "GET" is asking a lot already, so as soon as you move past the POST functions available in web forms you are pretty much writing a custom client to consume your web service. The approach is not RESTful, doesn't fit into large scale web architecture, and doesn't play nice with firewalls that don't expect these oddball methods.

My angle of attack is really from one of a controlled environment such as a corporate intranet or a control system in some kind of industrial infrastructure. The problems of large scale web architecture and firewalls are easier to control in this environment, and that's why CORBA has seen some level of success in the past and SOAP may fill a gap in the future. I'm not much of a fan of SOAP, and the opportunities that dealing with a function call as (method, resource, headers, body), or to my mind as (function call, object, meta, parameter data) are intriguing to me. Of particular interest is of how to deal with backwards and forwards-compatability of services through a unified name and method space and the ability to transmit parameter data and return "return" data in various representations depending on the needs and age of the client software.

I'm also interested in the whether the REST approach (or variants of it) can be scaled down to less-than-internet scale, and indeed less-than-distributed scale. I'm curious as to what can happen when you push the traditional boundaries between these domains about a little. I think it's clear that the traditional object model doesn't work on the Internet scale, so to my mind if we are to have a unified model it will have to come back down from that scale and meet up with the rest of us somewhere in the middle. I think the corporate scale is probably where that meeting has to first take place.

My suggestion is therefore that at the corporate scale a mix of restful and non-restful services could cooexist more freely if they could use HTTP directly as their RPC mechanism. Just a step to the left is actual REST, so it is possible to use it wherever it works. A step to the right is traditional Object-Orientation, and maybe that helps develop some forms of quick and dirty software. More importantly from my viewpoint it might force the two world views to acknowledge each other, in particular the strengths and weaknesses possessed by both. I like the idea that on both sies of the fence clients and servers would both be fully engaged with HTTP headers and content types.

I'm somewhat reticent to use a two-method approach (GET and POST only). I don't like POST. As a non-cachable "do something" method I think it too often turns into a tunneling activity rather than a usage of the HTTP protocol. When POST contains SOAP the tunnelling effect is clear. Other protocols have both preceeded and followed SOAP's lead by allowing a single URI to do different things when posted to based on a short string in the payload. I am moderately comfortable with POST as a DOIT method when the same URI always does the same thing. This is effectively doing the same thing as python does when it makes an object callable. It consistently represents a single behaviour. When it becomes a tunnelling activity, however, I'm less comfortable.

Robert, you mention the activity of firewalls in preventing unknown methods passing through them. To be honest I'm not sure this is a bad thing. In fact, I think that hiding the function name in the payload is counter-productive as the next thing you'll see is firewalls that understand SOAP and still don't allow unknown function names passing through them. You might as well be up-front about these things and let IT policy be dictated by what functionality is required. I don't think that actively trying to bypass firewalling capabilities should be the primary force for how a protocol develops, although I understand that in some environments it can have pretty earth-shattering effects.

In the longer term my own projects should end up with very few remaining examples of non-standard methods. As I mentioned in the earlier post I would only expect to use this approach where I'm sending requests to a gateway onto an unfixably non-RESTful protocol. REST is the future as far as I am concerned, and I will be actively working towards that future. This is a stepping-off point, and I think a potentially valuable one. The old protocols that HTTP may replace couldn't be interrogated by firewalls, couldn't be diverted by proxies, and couldn't support generic caching.

Thanks to Peter Hardy for your kind words also. I'll be interested to hear your thoughts on publish/subscribe. Anyone who can come up with an effective means of starting with a URI you want to subscribe to and ending up with bytes dribbling down a TCP/IP connection will get my attention, especially if they can do it without opening a listening socket.

Benjamin

I first became aware of the existence of Zeroconf some years back as a system for assigning IP addresses in the absence of any centralised coordination such as a DHCP server. It seemed designed for small networks, probably private ones. I already had my own setups in place to cover this kind of configuraton, so my interest was minimal. I've just stumbled across Zeroconf again, this time in the form of Avahi, a userland interface to zeroconf features. Even then, I didn't pay much attention until I hit Lennart Poettering's announcement that GnomeMeeting has been ported to Avahi. The mandatory screenshot shook me out of my assumption that I was just looking at a facility to assign IP addresses. The most important feature of Avahi is service discovery, and all using our old friend the SRV record combined with Multicast DNS (mDNS).

Well, it's nice to know that other people out there are trying to solve the same problems as you are. It's even nicer to hear they're using standard tools. The DNS Service Discovery page contains a bunch of useful information and draft proposals to try and solve just the problems I have been thinking about in the service discovery sphere.

For the kind of infrastructure I work with the main requirement of service discovery is that I can map a name not just to an IP address but to a port number. That's solved by SRV records, so long as standard clients and libraries actually use them. The main architectural contraint I work with is a requirement for fast failover once a particular host or service is determined to have failed. A backup instance must quickly be identified and all clients change over to the backup as soon as possible, without waiting for their own timeouts. This seems to be partially addressed by pure mDNS, as changes to the DNS records are propageted to clients immediately via a multicast UDP message. Unfortunately such messages are unreliable, so it is possible that an portion of the network from zero clients to all clients will miss hearing about the update. Polling is required to fill in the gaps in this instance. Alternatively, the DNS-SD page points to an internet-draft that specfies a kind of DNS subscpription over unicast. This approach parallels my own suggestions about how to add subscription to the HTTP protocol. In fact, it would be viatally important to be able to detect server failure efficiently whenever HTTP subscription was in operation and a backup available. If no such mechanism was in place any HTTP subscription could silently stop reporting updates with noone the wiser that their data was stale.

The parallels between DNS and HTTP are interesting. Both are based on fetching data from a remote host through a series of caches. Both have evolved over the years from simple request/response pairs towards long lived connections and pipelining of requests and responses. I hope that this new DNS-subscribe mechanism gets off the ground and can clear the way for a later HTTP subscription mechanism based on the same sorts of approach. Another hope of mine is that the Avahi project is successful enough to force software developers to make service discovery an issue or be trampled in the herd. The DNS work in this area is mostly pretty good and certainly important to people like me. The server side supports SRV, but the clients are still lagging behind. Additionaly, interfaces that should be appliable to SRV like the getaddrinfo but are not yet using SRV records should be updated where possible. A file should be added to /etc (perhaps /etc/srv) as part of the name resoloution process to support non-DNS use of these facilities without changing code.

I feel that one of the reasons that SRV has not been widely adopted is the complexity not in the initial lookup, but in the subsequent behaviour required after a failed attempt to contact your selected server. Clients are required to try each entry in turn until all fail or they find one that works. In terms of pure software design, it can be hard to propagate an error that occurs in trying to connect to a service back to the code that performed the name lookup to do so. You have to store state somewhere as to which you've tried so far. That's tricky, especially when your code dealing with DNS and service lookup is already horribly complicated. I don't really know whether this kind of dynamic update for DNS would make things better or worse. In one sense things could be more complicated because once DNS says that a service is failed you might want to stop making requests to it. On the other hand, if DNS can tell you with reasonable accuracy whether a particular instance of the service is alive or dead you might be able to do without the state that SRV otherwise requires. You could work in a cycle:

1. Lookup name. Get a non-failed instance based on SRV weightings.
2. Lookup service. If this fails either report an error, or go back to (1) a fixed number of times

There is obviously a race condition in this approach. If the service fails after the initial lookup or the notification of failure doesn't propagate until after the lookup occurs then looking up the service will fail. Alternatively, a service instance might be restored but still appear to be failed and thus noone will try to connect to it. Additionally, these constant failed and not failed indications propagating across the internet could have detrimental effects on bandwidth. On the other hand, they may prevent more expensive HTTP requests that would otherwise take place. Also, this is not the approach you would take with subscription. Preferrably, you would keep constant watch over whether your selected service instance was still within the living set and whenever it ceased to be so drop your subscriptions and select a new instance of the service. This may add complexity back into the chain, but I still think that relying only on the DNS end of things to determine who you talk to rather than a combination of DNS and the success or failure of attempts to actually reach a service instance would make things simpler overall.

I think this would be an interesting area for further research. I have good feelings about how this would work on a small network with multiple services per host and with multiple instances of your services spread around the network. There may also be benefits at the internet level. Chances are that unless there are internet-scale advantages we'll continue to see existing client libraries and programs failing to really engage with the idea.

Benjamin

I've been delving pretty deeply into HTTP lately, implementing such things in a commercial environment mostly from scratch. Interfacing to existing systems with a REST leaning is interesting, and when you're viewing things from the low levels without reference to SOAP and other things that have come since HTTP you get to see what HTTP is capable of out of the box. In particular, I think you get to see that even if you aren't a RESTafarian SOAP is probably the wrong way to approach applying functions to HTTP-accessable objects.

Standard methods

HTTP is built around some standard methods, and a subset of these are the classic methods of REST. REST proponenents usually try to confine themselves to GET, PUT, DELETE, and POST. rfc2616 specifies OPTIONS, GET, HEAD, POST, PUT, DELETE, TRACE, and CONNECT. Different versions of the specification have different method lists. rfc1945 covering HTTP/1.0 only includes the GET, HEAD, and POST methods. The earlier HTTP/1.1 rfc 2068 didn't include CONNECT but did include descriptions for additional PATCH, LINK, and UNLINK methods. WebDAV adds PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, and UNLOCK. REST is really based not around the four traditional verbs but the total defined set of methods that are commonly understood. The use of anything that is used and standardised and agreed upon is RESTful. If we go back to the Fielding dissertation we read the single sentence that sums up REST:

REST enables intermediate processing by constraining messages to be self-descriptive: interaction is stateless between requests, standard methods and media types are used to indicate semantics and exchange information, and responses explicitly indicate cacheability.

So long as we use standard methods we are being RESTful, and standard methods are any that have been agreed upon. Methods that have been agreed upon can be processed by caches in standard ways, and making caches work is the most significant architectural constraint that goes into defining the REST approach.

In fact, extensions can be added arbitrarily to HTTP. Any method not understood by a proxy along the way causes the proxy to mark the affected resource as dirty in its cache but otherwise to pass the request on towards an origin server. I think the simplest way to map function calls onto HTTP resources is to use the function name (or a variant of it) to the HTTP method rather than including it in the HTTP body. This is made a little tricky by two factors. The first is that if anyone ever does come along and define a meaning for your method then caches might try and implement that meaning. If you're lucky they'll still only treat it as a cache clearing operation and a pass-through. On the other hand you might not be lucky. Also, new clients might come along expecting your method to behave according to the new standard semantics and cause further problems. Methods are effectively short strings with universal meaning. Dan Connolly has this to say:

There is a time and a place for just using short strings, but since short strings are scarce resources shared by the global community, fair and open processes should be used to manage them.

So to make the best of it we shouldn't be using methods that might clash with future meanings. I suggest that using a namespace in the HTTP request method would eliminate the possiblity of future clashes and make the HTTP method something more than the wrapper for a function call. It can be the function identifier.

URIs as methods, giving up "short strings"

The second issue arises from the first: Exactly how do we do this? Well, according to rfc2616 any extension method just has to be a token. It defines a token as any character except for those in the string "()<>@,;:\\\"/[]?={} \n\r". This rules out a normal URI as a method. The URI would at least need to be able to specify slash (/) characters to separate authority and path. I suggest a simple dotted notation similar to that of java namespaces would be appropriate. Now we are no longer being RESTful by heading down this path. We aren't using standard methods anymore. On the other hand, perhaps it is beneficial to be able to mix the two approaches every once in a while. Perhaps it would help us to stop overloading the POST method so badly to make things that don't quite fit the REST shape work.

Practical examples, and the spectrum of standardisation

Your new method might be called ReloadConfiguration.example.com. It might even be tied to a specific class, and be called ReserveForExecution.control.example.com. If your method became something that could be standarised it might eventually be given a short name and be included in a future RFC, at which time the RFC might say that for compatbility with older implementations your original dotted name should be treated as being identical to the short name.

Mostly-RESTful? Mixed REST and non-REST.

I think that author James Snell largely gets it wrong in his IBM developerworks article on why both REST and SOAP have their places. I think his view of REST is a bit wonky, but to lift things up to a more abstract level I think he has some interesting points. He seems to think that some operations can't easily be translated into state maintained as resources. He thinks that sometimes you need to trigger activities or actions that don't work in a RESTful way. Mostly, he want to be able to employ traditional gof-style OO patterns to his distributed architecture. I don't find myself agreeing with any of his examples but I do find myself in trying to retrofit REST onto an existing architecture not wanting to take the big hit all at once. What I want is a migration path. Even when I get to the end of that path I think there will still be some corners left unRESTful. There are places where one application is already a converter to another protocol. That protocol isn't RESTful, so it seems unlikely to me that putting a RESTful facade over the top of it will ever do any good. That protocol is also an industry standard, so there's no chance of tweaking it to be more like we might like in our RESTful world.

What I'm suggesting is that by supporting non-RESTful concepts alongside RESTful ones it becomes possible to shift things slowly over from one side to the other, and find the right balance where the fit isn't so good for REST.

Once you start heading down this path you see some interesting features of HTTP. After allowing you to select the function to call and the object to call it on HTTP allows you to provide a body that works like the parmaeter list, and extra metadata which you might use along the way in the form of headers. Now that we have cleared the method identifier out of the body it can be decoupled from its content. We can effectively have polymorphism based on any of the metadata headers, such as content-type. We can handle multiple representations of the same kind of input. We can also introspect sophisticated content types such as XML to determine how to do further processing. In my case I found that I had to map onto a protocol that supported string positional parameters. For the moment the simplest fit for this s csv content. Just a single CSV line supports passing of positional parameters to my function. In the future as the underlying protocol changes shape to suit my needs I expect to support named parameters as an XML node. Eventually I want to push the HTTP support back to the server side, so any resource can decide itself how to handle content given to it.

Conclusion

I hope that HTTP will support future software revisions that are more and more RESTful into the future, but for the moment can concentrate on supporting necessary functionality without a massive flow-on impact.

Benjamin

I just read these articles published by Ryan Tomayko's lesscode site on disposable software and the fallacy of resuse. They immediately strike a chord with me. Especially when you're talking about inexperienced software developers it is easy to go down the path of overengineering with a justification of "it'll make implementation of the next set of requirements easier" or "it'll make it more generic". The last few years of agile development methodologies have been working to belie these arguments. Indirection is not abstraction.

When you write more code to do the same thing,
when you put a wrapper in front of a perfectly servicable abstraction or concrete class,
when you hide what is really going on in order to make your code "higher level",
when you write software you don't have requirements for yet,
when you prepare the way for the coming king,
when you code a generic design before you have two working concrete designs and three sets of concrete requirements,
you make it harder to understand,
you make it harder to maintain,
you make it more costly both now and into the future.

Software is like a user interface. The fewer concepts your users have to be aware of the more likely they'll be able to write software that uses yours. The fewer concepts you have in your code the more likely someone else will be able to review and maintain it.

Reusuable software is not a valid goal in and of itself because in the very act of creating a great reusable component you introduce another concept into your software, perhaps more than one. Your software is worse because of it. Unused reusability is actively harmful. Design what you need and go no further.

Your software is not your baby. It is not something you can hang onto. Be ready to throw it away and start again. The worth of your software is not in your software. The worth of your software is in you. The intellectual property of the company you work for is not what they gave you thousands of dollars to build. It is what they gave you thousands of dollars to become.

Benjamin

I've just come across this May article by Michael Ellerman. Michael responds to some inflammatory language I used in my own response to a Joel Spolsky article. I obtusely referred to "real work" and my doubt that python is suited. Both my article and Michael's are centred around the concepts of type and of Design By Contract. I have since written about type in the context of the REST architectural style.

Compile time or runtime?

My feelings are still mixed about the use of explict type hierarchies. On one hand I have years of experience in various languages that show me how type can be used by the compiler to detect bugs even when no test code has been written, or when the test code doesn't achieve 100% coverage of the input space. That capability to detect error is important because coverage is theoretically impossible for most functions. The gap between what is tested and what is possible must be covered to give any confidence that the code itself is correct. Stop-gaps have traditionally come either in the form of manual code review or automatic type-supported compiler checks. On the other hand, it is clear that even a carefully-constructed type hierarchy isn't capable of detecting all contract breeches at compile time. Function preconditions must still be asserted, or functions must be redefined to operate even in cases of peculiar input. The assertions are run time checks in languages I'm familar with, and it may not be possible to convert all of them to compile time checks.

Valid and invalid function domains

It should always be possible to define the valid input space to a function. Proving that the actual input is a member of the set of valid input is typically harder. Type hierarchies are a tool to tell the compiler that the appropriate reasoning has been done by the time it reaches the function. Unfortunately, the complexity of a type system that could effectively define the domain of every function in your program is so great that implementing things this way may cripple your programming style. You would definately be taking a step away from traditional procedural approaches and moving firmly towards a mathematical or functional view of your application. These world views have traditionally been more difficult for developers to grasp and work effectively in, except perhaps in the especially clever among us.

The other thorn in the side of type systems is actual input to the program. It typically comes from a user, or perhaps a client application. The process of converting the data from raw bytes or keystrokes into useful input to a complex typing system can be quite tricky. Your program has to examine the data to determine which type best reflects its structure, and this must take into account both what it is and how it will eventually be used. It is trivially provable that any function that accepts unconstrained input and tries to construct objects who's types match the preconditions of another function must be runtime checked for as far as the unconstraned input is a superset of the legal constrained input. Because it is important to know how data will be used before using it these runtime-checked functions are likely to appear throughout your actual program. The more I/O connected your program is, the more you will have to deal with runtime checking.

Bucking the type hierarchy

I believe that REST and a number of other approaches which reject planned type hierarchies are founded in this area where handling of data from outside the program is difficult or impossible to isolate to a small part of the program's code. When this is the case the use of formal typing may not add any value. Just apply the same paradigm everywhere: Deal with a very basic fundamantal type and allow the data to be rejected if required. Handle the rejection in a systematic way. To me, the use of exceptions is a hint that this approach was brewing in headline Object-Oriented languages long before the pythons and RESTs of this world found a mass appeal. Exceptions are thrown when input to a function is not with the function's domain, but when that condition is not necessarily an indication of error within the program. If the error occured because of faulty input outside the program's control it makes sense to throw and catch exceptions so that the original input can be rejected at its source. The alternative would be to deal with the error wherever it occured or to specifically structure your program to include error-handling modules or classes. Within the classical strongly-typed O-O system without concerns for external input exceptions are evil, but when used to handle faulty external input consistently they can be a force for good.

The SOAP Intervention

Like CORBA, SOAP attempts to extend the Object-Oriented programming model across programs. This allows servers to treat input data not as streams of bytes but as fully formed structures and object references with their place in the type hierarchy already assigned. Various attempts have come and gone to actually serialise and deserialise objects between processes. These approaches should make it possible to eliminate significant network-facing handling of very basic types. Now, if only we could make the user start entering serialised objects as well... ;)

These approaches are interesting. They fail to some extent because of the assumptions they tend to make about network latency being small. They fail somewhat because they don't adequately support concepts of caching and of redirecting requests. On the other hand, I think there is some evidence these days that they fail simply because they have a system formal type hierarchy. If a programming language is like any other program, it benefits from being able to do what the user wants without the user having to understand too many concepts. These systems are built around the idea that the large organised way we have written software in the past is the correct one, and that all we need to do is expose this programming model on a larger scale. I think it was the success of Visual Basic that first saw this world view really shaken. It turns out that when you allow the average bloke off the street to write software the software he writes is pretty simple. It may have less stringent performance and safety requirements to what we professionals believe we have. It may be smaller and less sophisticated than we tend to come up with. On the other hand, it works. It does a job that that guy certainly wouldn't have hired us to do for him. It scratches an itch and empowers him. I think there's some evidence that more software will be written in the future at the end of the scale than at the end we're used to working from. In fact, I suspect that one day we'll all be writing software using the same mental models that Joe Bloggs works with and we'll find that we can still do the good job we think we're doing now using those approaches.I think there's some evidence that more software will be written in the future at the end of the scale than at the end we're used to working from. In fact, I suspect that one day we'll all be writing software using the same mental models that Joe Bloggs works with and we'll find that we can still do the good job we think we're doing now using those approaches.I think there's some evidence that more software will be written in the future at the end of the scale than at the end we're used to working from. In fact, I suspect that one day we'll all be writing software using the same mental models that Joe Bloggs works with and we'll find that we can still do the good job we think we're doing now using those approaches.I think there's some evidence that more software will be written in the future at the end of the scale than at the end we're used to working from. In fact, I suspect that one day we'll all be writing software using the same mental models that Joe Bloggs works with and we'll find that we can still do the good job we think we're doing now using those approaches.

Conclusion

Michael proposes the use of assert isinstance(s, SafeString) to do type checking in python. I'm not sure this is really "good python", which normally focuses on what an object can do rather than what type you've been able to tag it with, but that's an aside. This isn't as useful as the type checking in C++ or Java because it is performed at runtime and only triggers when the input is not what was expected. He points out that C++ does no better on more complex contract provisions such as "i < 3", which must be made a precondition. My original point with respect to this issue is really about how far testing can take you on the path to program correctness. Using hungarian notation as Joel Spolski originally suggested may help a code reviwer determine program correctness, but in the python world there is no automatic genie to help you pick up those last few percentage of missed errors. In a large and complex program with incomplete test coverage my experience says that suitable compile-time type checking can find errors that code reviwers miss. On the other hand, if you're dealing with a lot of external input (and let's face it, we all should be these days) type may just not help you out at all. My feeling is that formal type hierarchies will eventually go the way of the cathedral as we just stop writing so much internally-facing code. My feeling is that we'll be much more interested in pattern matching on the input we do receive. Given python's use profile it is probably appropriate that input outside of function domains throws an exception rather than raising a runtime check, but in the domains I'm used to working in it doesn't seem that python is fundamentally different enough to or more advanced than the langauges we've seen in the past. Not so far advanced yet to warrant or make up for such a scaling back of tool support for compile-time checking. On the other hand I keep talking up REST concepts and the death of type to my own collegues. I guess you'd say that right now I'm still in the middle, and want to see more return for giving up my comfortable chair.

Benjamin

When characterising an interface within a distributed system I ask myself a series of questions. The first two I ask together.

1. Which direction is the data flowing, and
2. Who is responsible for making this interface work?

I also ask:

• What is the data model of the interface, and
• Which protocol do we speak?

The first is easy. It starts where the data is, and it ends where the data isn't. It's a technical issue. The second is more complicated and is focused around where the configuration lives and who is responsible for maintaining that configuration, especially configuration for locating the interface. The client is responsible for knowing a lot more than the server, although it may discover some of what it needs to know as it accesses the interface itself. Web browsers get all the information they need for navigation from links on a page, but even they need a home page or a manually-entered URL to get them going. Machine to machine interactions don't have the luxury of a human operator telling them which way to go so typically have fewer steps between their starting configuration and the configuration they use in practice to operate.

When data flows from client to server life is easy. The client can push, and if it's data doesn't get through it can just try again later. It can use bandwidth fairly efficiently even with the HTTP protocol. The use of pipelining allows the client to use available bandwidth efficiently so long as it is generaing an idempotent sequence of requests. Mixes of GET and PUT aren't idempotent (although each request itself should be), so they can cause stalls on the pipeline and reduce performance to a level that depends on latency rather than bandwidth. Depending on the client-side processing this may be able to be avoided altogether or contained in a single client function to avoid overall performance bottlenecks. This is important because it is easier to increase bandwidth than to reduce latency. Unfortunately it has something to do with the speed of light.

The problem is that data often flows both ways. You could reverse the client-server relationship depending on which way the data flows, and sometimes this is appropriate. On the other hand, you're sure to eventually need to push data from server to client. Today's Internet protocols aren't good for this. To reverse the client-server relationship in HTTP you need a HTTP server. That isn't hard. The hard part is opening the port required to accept the associated connections.

At present we have a two-tier internet developing. We have the high end established servers that can accept connections and participate in complex collaborations. We also have the low end of ad hoc clients trapped behind firewalls that will allow them to connect out but not allow others to connect back in. SOAP protocols are devised for the top tier Internet, and rely on two-way connectivity to make things work. I think ourt target should be the second tier. These clients are your everyday web browsers, and when data has to flow from a server to one of these web browsers we don't have any good established options open. Clients are reduced to polling in order to allow data to flow from server.

The GENA protocol I mentioned previously is built for the top tier internet. On the bottom tier the following constraints apply:

• We can't accept connections, only make them
• We must speak an established protocol, or get a new protocol very widely adopted
• We must work with existing proxies

That rules out everything so far proposed, I think. I have spent some time on this myself, though. I have a protocol that is regular HTTP and only connects out from the client. There are some other considerations I would like to see work:

• The subscription should be able to be handled by another server to the resource itself
• A subscription which is handled by the same server should take no more than one single request to establish
• The server should be able to make use of available bandwidth
• Overload conditions should be handled nonfatally (message dropping)
• Message ordering should be guaranteed, preferrably by TCP/IP

The whole thing should be RESTful, so

• The data being transmitted should be a representation the state of the resource (either as diff or whole value)
• The subscription should be modelled as a resource

Here is the closest I've been able to come up with so far:

• Client sends request to resource with SUBSCRIBE method
• Server creates subscription resource and returns in Location header of 201 "Created" with current value as body
• Client sends request to subscription with NEXT method
• Server responds with 200 and next data when it occurs
• Client sends request to subscription with NEXT method
• Server responds with 200 and next data when it occurs
• ...
• Client sends DELETE request to subscription, or gets 404 or 403 after server deleted it

The theory is that the subscription keeps track of whether new data is available at any time. When the NEXT request arrives it returns the data immediately if it is available. If new data isn't available it holds of replying until data is available. If a proxy is sitting in between client and server it would eventually time out causing the client to issue a new NEXT request.

Clearly this approach has problems. I think the creation of the subscription is fine, but the actual subscription has several problems. The first is that it can't make use of available bandwidth. This problem is endemic to the proxy behaviour and can't be solved without a change to the HTTP protocol that allows multiple responses to a single request rather than this single request/response pair. The second is that no confirmation is given back to the server. A response may be sent down the TCP/IP conection that the NEXT request arrived on but never be transmited to the client due to a connection being closed. This can be solved by adding a URI to both the NEXT request and the two responses available for the NEXT and SUBSCRIBE requests. As an alternative, NEXT requests may have to be directed to the URI(URL) of the next value rather than being sent to the subscription. Responses would have to specify a URI that should be used in the next NEXT request passed to the subscription. If the URI matches what is currently available the server should return the data immediately with a new URI. If the URI doesn't matches an older state the server should return the state, but also indicate how many updates were missed (if possible) back to the client. If the URI is still a future URI (the next URI) the response should be deferred.

The proxies get in the way of a decent solution. Really, the only solution is to come up with a new protocol (perhaps a special extension of HTTP) or use a different existing protocol.

At least one person I know likes to talk about Jabber whenever publish-subscribe comes up. Here is the standard defined for XMPP. The thing that immediately gets the hairs at the back of my neck going is that Jabber doesn't seem to immediately support REST concepts. It's a message passing system where the conceptual framework relies on you connecting to someone who knows more than you about how to find things on the network. That doesn't seem right to me. I prefer the concept of caches that mirror network topology to the idea of connecting to some server three continents away that might be arbitrarily connected to some other servers but most probably is not connected to everything, just a subset of the Internet. My thinking also leads me to think of publish-subscribe as an intrinsic part of a client-server relationship rather than this thing that you slap onto the top of either an established HTTP protocol or an established XMPP protocol.

Those things said, oh gosh it's complicated. I really think that you don't need much over existing HTTP to facilitate every possible collaboration technique, but as you enter the XMPP world you are immediately hit with complicated message types combined with HTTP return codes combined with more complicated return nodes. The resource namespaces don't seem well organised. I'm not sure I quite understand what the node "generic/pgm-mp3-player" refers to in the JEP's example. It's all peer to peer rather than client server and... well... I'm sorry that I can't say I'm a fan. Maybe once it's proven itself a little more I'll give XMPP another look.

I've already suggested some more radical approaches to adding subscription support to HTTP. I do believe it's a first class problem of an internet scale protcol and should be treated as one. I think that making appropriate use of available bandwidth is an important goal and constraint. Unfortunately, I believe that working with the existing Internet infrastructure is also important. At the moment proxies make this a hard problem to solve well. In the interim, feel free to try out my "NEXT" HTTP subscription protocol and see how you like it. It may at least open things up to the second tier of users.

Benjamin