Sound advice - blog

Tales from the homeworld

My current feeds

Mon, 2009-Aug-31

WADL for REST-style SOA

In the words of Mark Nottingham:

CORBA has IDL; SOAP has WSDL. Developers often ask for similar capabilites for HTTP-based, "RESTful" services. While WSDL does claim support for HTTP, isn't well-positioned to take advantage of HTTP's features, nor to encourage good practice.

is designed to fill this gap. It is HTTP-centric and attempts to provide a straightforward syntax for describing the semantics of particular methods on resources.

Resources and SOA

Resources are a key concept in , and also in REST-style SOA. A service expresses its interface as a set of resources. All resources share the same Uniform Contract. However, different resources have different associated semantics.

Resources effectively replace the traditional service-specific contract in a . In doing so they introduce a meta-data gap. Where the contract previously described the interface to the service in a single coherent place, the Uniform Contract of resources does little to describe the interface of a given service.

Filling this gap is an important part of applying the REST style to SOA. This occurs in two parts: One part is the application of additional hypermedia so that clients can locate the correct resource to invoke requests upon based on the information that they are likely to have at hand. This hypermedia is incorporated into the Uniform Contract in terms of defined link relationship types and dedicated hypermedia-intensive media types. The second part is the context where WADL can be applied, and is incorporated into the service description.

Objectives of WADL in SOA

The kind of machine-readable description WADL could offer is required to fulfil a number of specific needs:

The information is not fundamentally for service consumer consumption beyond a basic level of discovery. Importantly, knowledge of relationships between resources should not normally be known by service consumers ahead of time. Consumers should make use of links between resources to discover relationships. Failure to adhere to this principle undoes a number of REST features such as the ability to link freely from one service to another with confidence that service consumers will successfully discover and exploit the specified resource at runtime.

Key Resource Metadata

I tend to look on the kind of interface description needed at this level as more of a table than a tree. I think that it is generally advisable to include most of the path when describing the semantics of methods on a given resource. Nevertheless, XML is adequate to describe this structure. I would tend to include the following features:

An example description in table format might be:

Base URLhttps://invoice.example.com
Resource Identification Method (Uniform) Media Types (Uniform) Cache Documentation
/{invoice id} GET
  • application/vnd.com.example..invoice+xml
  • application/vnd.com.visa..invoice+xml
Must revalidate Retrieve invoice for invoice id
PUT
  • application/vnd.com.example..invoice+xml
No cache Update Invoice for invoice id to match specified state
/{invoice id}/payment/ POST
  • application/ebxml.transaction+xml
No cache Add a payment relating to this invoice
/{invoice id}/payment/{trans id} GET
  • application/ebxml.transaction+xml
5 minutes Fetch a specific transaction for this invoice
/?[date=]&[customer=]&[paid=] GET
  • application/vnd.com.example.invoice+xml
Must-revalidate Query for invoices with specified properties

Each line describes a set of resources corresponding to the URL template. The template is filled out with parameters that the server will interpret when the request is processed. URLs can be seen as a message from the service to itself. It should not generally be parsed outside of the service, nor constructed outside of the service. I tend to use the query convention in the last URL template to indicate this rule is being broken and explicit service/consumer coupling is being introduced.

Methods and Media Types are referred to by their identifier only. There is no need to include them in a service description, because they should already be adequately described in uniform contract specification documents. Supporting multiple media types for a given method is important in an architecture with an evolving set of media types. It allows old services and consumers to continue to interact with new ones over the transition period without having to perform simultaneous upgrade.

Cache is also important, as this is a key REST constraint that needs to be described to support governance activities in compliance with the constraint.

While nesting exists to a point, there is no strongly-implied relationship between different nested or non-nested URLs. Each resource has its own distinct semantics. Hypermedia is incorporated into multiple resources by way of links embedded in invoice representations. For example, invoices are likely to include a link to the customer entity that the invoice was made out to. The set above also includes hypermedia in the form of query URLs that a service consumer who has a number of starting parameters can use to find the invoices they want.

Applying WADL to the problem

The WADL equivalent for the above service metadata is follows:

<application
	xmlns="http://research.sun.com/wadl/2006/10"
	xmlns:html="http://www.w3.org/1999/xhtml"
	xmlns:xsd="http://www.w3.org/2001/XMLSchema"
	>
	<doc>
	<html:p>Meta-data for the Invoice service, corresponding to the
	<html:a href="https://example.com/interface-control/resource">
		example.com uniform interface</html:a>
	</html:p>
	</doc>
	<resources base="https://invoice.example.com">
		<resource path="/{invoice-id}">
			<param name="invoice-id" style="template" type="xsd:NMTOKEN"/>
			<method name="GET">
				<doc><html:p>Retrieve invoice for invoice-id</html:p></doc>
				<response>
					<representation mediaType="application/vnd.com.example..invoice+xml"/>
					<representation mediaType="application/vnd.com.visa..invoice+xml"/>
				</response>
			</method>
			<method name="PUT">
				<doc><html:p>Update Invoice for invoice id to match specified state</html:p></doc>
				<response>
					<representation mediaType="application/vnd.com.example..invoice+xml"/>
				</response>
			</method>
		</resource>
		<resource path="/{invoice-id}/payment/">
			<param name="invoice-id" style="template" type="xsd:NMTOKEN"/>
			<method name="POST">
				<doc><html:p>Add a payment relating to this invoice</html:p></doc>
				<response>
					<representation mediaType="application/ebxml.transaction+xml"/>
				</response>
			</method>
		</resource>
		<resource path="/{invoice-id}/payment/{trans-id}">
			<param name="invoice-id" style="template" type="xsd:NMTOKEN"/>
			<param name="trans-id" style="template" type="xsd:NMTOKEN"/>
			<method name="GET">
				<doc><html:p>Fetch a specific transaction for this invoice</html:p></doc>
				<response>
					<representation mediaType="application/ebxml.transaction+xml"/>
				</response>
			</method>
		</resource>
		<resource path="/">
			<param name="date"
				style="query" required="false" type="xsd:dateTime"
				/>
			<param name="customer"
				style="query" required="false" type="xsd:anyURI"
				/>
			<param name="paid"
				style="query" required="false" type="xsd:boolean"
				/>
			<method name="GET">
				<doc><html:p>Query for invoices with specified properties</html:p></doc>
				<response>
					<representation mediaType="application/ebxml.transaction+xml"/>
				</response>
			</method>
		</resource>
	</resources>
</application>

That actually wasn't too painful. It was easy enough to mimic the table structure. I think this makes the description of a resource more readable. It handled the various parameters to these URLs in a straightforward way. The only thing really missing from this description is the caching information from our earlier table.

Conclusion

I think WADL is more or less suitable as a machine-readable media type for describing the set or resources exposed by a service. It could perhaps do with some extensions (and better extensibility), but it seems like a good starting point to me.

I have written about WADL from a slightly different perspective previously.

Benjamin

Sun, 2009-Aug-16

MIME types holding REST back

With the increasing focus on within enterprise circles, the practice of how REST gets done is becoming more important to get right outside of the context of the Web. A big part of this is the choice of application protocol to use, the "Uniform Contract" exposed by the resources in a given architecture. Part of this problem is simple familiarisation. Existing enterprise tooling is built around conventional RPC mechanisms, or layered on top of HTTP in SOAP messages. However, another part is a basic functional problem with HTTP that has not yet been solved by the standards community.

HTTP/1.1 is a great REST protocol. GET and PUT support content negotiation and redirection. They are stateless, and easy to keep stateless. They support layering. GET fits extremely well into caching infrastructure. These methods fit into effective communication patterns that solve the majority of simple distributed computing communication problems. HTTP works well with the URI specification, which remains best practice for identifying resources. HTTP also accommodates extension methods in support of its own evolution, and in support of specialisations that may be required in specific enterprise architectures or service inventories.

A significant weakness of HTTP in my view is its dependence on the standard for and on the related iana registry. This registry is a limited bottleneck that does not have the capacity to deal with the media type definition requirements of individual enterprises or domains. Machine-centric environments rely in a higher level of semantics than the human-centric environment of the Web. In order for machines to effectively exploit information, every unique schema of information needs to be standardised in a media type and for those media types to be individually identified. The number of media types grows as machines become more dominant in a distributed computing environment and as the number of distinct environments increases.

Media type identification is used in messages to support content negotiation and appropriate parser or processor selection. At the scale of the Web, only a small number of very general types can be accommodated. It is difficult to obtain universal consensus around concepts unless the concepts themselves are universal and agreeable. Smaller contexts, however, are able to support a higher degree of jargon in their communication. An individual enterprise, a particular supply chain, a particular problem domain is capable of supporting an increased number of media types over and above the base set provided by the Web. The ability to experimentally define and evolve these standards outside the Web is essential to a healthy adoption of the REST style and related styles beyond the Web.

An example of the capability to perform media type negotiation with HTTP can be found in the upgrade from RSS to ATOM feeds. While the human element of this upgrade rarely required this in practice, HTTP makes it possible for a client to state which of these types it supports. The server is then able to respond with content in a form the client understands. In a machine-centric environment, this is even more important. Few content types used in the early days of most architectures will survive into maturity. Types will change and evolve, and many will be superseded. Machine-centric environments do not have the same capability to change URLs based on their upgrades, so content negotiation based on media type allows incremental upgrade of a system... one component at a time.

A URL-based Alternative

The main alternative for media type identification would be to use . These already provide a decentralised registry, and can double at the URL of the related human-readable specification document. These seem like a simple answer. Existing IANA types can be grandfathered into the scheme with a https://www.iana.org/assignments/media-types/ prefix, which would already point as a URL to an appropriate specification document for a number of types.

However, URNs suffer three problems when compared to MIME identifiers. The first is simply that HTTP does not permit their use in the appropriate headers. The second is that URNs cannot be further interpreted when they are read. The third is that URNs cannot be parameterised as MIME types are.

MIME types are capable of identifying not only their specific type, but additional types they inherit from or are interpretable as. For example, most XML media types include an extension "+xml". This allows generic processors of XML to interpret the content based on broad-brushed generic mechanisms. One could extend this concept to support specialisations of higher-level media types such as atom. Storing a specific structure of data within an atom envelope does not prevent it from being interpreted as atom. Leaving this information in place within the media type identifier gives generic processors additional visibility into the messages being exchanged.

The use of parameters on media types would essentially not be possible within the URL itself. These could be included in syntax around the media type if they are still required. Typically, XML and binary types should no longer require these headers... so this may be of historical and plain text importance only. Plain text types will often be able to use different HTTP fields to convey relevant information such as their content-encoding.

The solution to these problems could be a revision of HTTP to include URI syntax support for media types, combined with a protocol whereby processors could determine whether one media type inherits from another. Whether HTTP can be revised is a difficult question to answer, but a protocol for discovering inheritance relationships is relatively easy to develop. One could either make use of HTTP headers in the GET response for the URI, or specify a new media type for media type description. The obvious approach with link headers would be to say Link: rel="inherits". However, this is a limited approach. An actual media type description media type could take the form of a short XML document or simple microformat for human-readability, and is perhaps more general and future-proof.

Specific changes that would have to occur in HTTP would apply to the Content-Type and Accept headers. Content-Type would be relatively easy to extend to support the syntax, however problems may emerge in any deviation from the definition of MIME itself and the use of this header within SMTP and other contexts. Accept, as well, would be relatively easy to extend. Quote characters such as " (&quote;) would need to be included around URLs to avoid confusion when ";" (semicolon) and "," (comma) characters are reached during parsing. This may impact backwards-compatibility.

Backwards-compatibility is a prime concern for HTTP. It may be worth doing some trials with common proxies and intermediaries to see how Content-Type and Accept header changes would impact them, to see just how big this problem would be in practice.

A decentralised registry in MIME syntax

An alternative to going all the way to URNs might be to make use of the DNS system alone as part of a media type identifier. For example, a "dns" sub-tree could be registered under MIME. The sub-tree would allow an example organization to safely maintain a set of identifiers beginning with application/dns.com.example without IANA coordination. Any organization with a presence in the DNS tree could do the same.

The main upside in this approach is that consistent syntax could be maintained for new media type identifiers. HTTP could be used as-is, and individual organizations could create, develop, and market their standards as they see fit. The "+" syntax could continue to be used for inheritance, so we might still end up with hybrid types such as application/dns.com.example.record+atom+xml. If this got out of hand we could be talking about fairly long media types, but hopefully the social pressures around reasonable media type identification would work against that outcome.

Perhaps the strongest argument against this alternative is a loss of discovery of standards documentation. URLs can easily be dereferenced to locate a specification document. This hybrid of DNS and MIME would need additional help to make that so. It would be necessary to have a means of translating such a MIME identifier into a suitable URL, which quickly leads into the world of robots.txt and other URL construction. While this is not a desirable outcome, at least it doesn't leave the lookup of a URL as an integral part of the parsing process. The URL-based solution may do that.

As a strawman, one might suggest that any MIME type registered in this way would be required to have something human-readable at a /mime path under the DNS name: e.g. application/dns.com.example.record would become the https://record.example.com/mime URL. This would be quite awkward.

A Hybrid Approach

A third alternative would be to define a way to encode URLs or a subset of possible URLs into media type identifiers. In this example the IANA subtree might be "uri" instead of "dns". The type name would have to be constructed so that the end of the dns part of the identifier could be found and further "." characters treated as URL path delimiters. For example, application/uri.com.example..record+atom+xml could indicate that the type can be interpreted as both xml and atom+xml. In addition, the specific specification for this variant of atom+xml can be found at <https://example.com/record>.

Specification Persistence

All decentralised options share possible persistence problem. We can probably trust the IETF to hold onto standards document for historical reference for a few hundred years. Can we trust a small start-up business to do the same? What happens when the domain name of an important standard is picked up by domain squatters? Most standards should still not be registered under an individual company name once they reach a certain level of importance. They should fall under a reputable standards body with a fairly certain future ahead of them.

Conclusion

I'm torn on this one. I don't want to go to the IANA every time I publish a media type within my enterprise. I like URLs, but want a straightforward way to discover important inheritance relationships. I don't want to break backwards-compatibility with HTTP, and there is no better protocol available for the business of doing REST. What's a boy to do?

My preference is to go with a hybrid approach. This would yield compatible syntax with today's HTTP, but still support a highly decentralised approach. Over time, a significant number of standards should climb the ladder from early experimental standard through to enterprise and industry standards to eventually become part of a universally-acceptable set that describe the semantic scope of the machine-readable Web. Their identifiers may change as they make the transition from rung to rung so that the most important standards are always under the namespace of the most well-respected standards bodies.

Benjamin