Hello World in Hypermedia Style

After perhaps reading too much about Hypermedia (too much hype?) I started getting the itch to do something. This quick blog is a record of the steps I took towards a Hypermedia “hello world”. The main tool I used was a plug-in REST client from Mozilla but anything that lets you set headers and inspect responses will do the job.

For this exercise I choose three popular Hypermedia designs: HAL, Collection+JSON and Siren. Each design has a high-level of abstraction making it suitable for re-use and an active community of developers who are busily engaged with various implementations.

HAL

In the words of Mike Kelly, HAL’s creator :

HAL is a format you can use in your API that gives you a simple way of linking

See more of the HAL specification. HAL is available in either JSON or XML and has a registered media-type of

application/hal+json

Mike has documented the HAL design by providing an interactive HAL Browser. This is the most immediate way to say “hello world” in Hypermedia style. Firstly you use the HAL browser to create an account (hint: see the hints :-)). Next you should navigate to /users/:account and then use the NON-GET button to get into the following dialogue box.

Saying it
Saying Hello World

It’s okay if you type something other than “hello world” *yawn* but changing anything else might break something .. To verify your creation navigate to the latest posts and as-if-by-magic your entry will appear (fingers x).

The HAL browser is great way to quickly get a feel for what HAL is all about. Armed with this knowledge I wanted to take a step towards a HAL client that could be controlled independently, and ultimately deployed in another context. The HAL browser seemed tightly coupled to the server, so rather than unpick it, I dug around and found another server running on the HAL builder site. I made the following request using my trusty REST client.

GET http://gotohal.net/restbucks/api
Accept: application/hal+json

And in case you weren’t paying attention earlier, here’s a picture.

Image

The HAL builder dutifully obeyed my request (yes, the Accept header is necessary) and gave me the following response.

{
    "_links": {
        "self": {
            "href": "/restbucks/api"
        },
        "orders": {
            "href": "/restbucks/api/orders"
        }
    },
    "name": "RestBucks Store",
    "project": "restbucks",
    "version": "0.0.1"
}

Now if I was a robot, I would crack-on and apply some automation to the above. Here are our first clues about Hypermedia. Custom media-types and machine-based connections! There’s way more discussion about using HAL in the  forum.

Collection+JSON

Mike Amundsen is the author of Collection+JSON and says it is

a JSON-based read/write hypermedia-type designed to support management and querying of simple collections

And just like HAL, Collection+JSON has its very own registered media-type. To get a test response from Collection+JSON we’ll return to our REST client. As before set the client up for the target service and specify the media-type.

GET https://graphviz-ewf.rhcloud.com
Accept: application/vnd.collection+json

This Hypermedia server responds with the following.

{
  "collection": {
    "version": "1.0",
    "href": "http://graphviz-ewf.rhcloud.com:80/",
    "links": [
      {
        "href": "http://graphviz-ewf.rhcloud.com:80/",
        "rel": "home",
        "prompt": "Home API"
      },
      {
        "href": "http://graphviz-ewf.rhcloud.com:80/graph",
        "rel": "graphs",
        "prompt": "Home Graph"
      },
      {
        "href": "http://graphviz-ewf.rhcloud.com:80/register",
        "rel": "register",
        "prompt": "User Register"
      },
      {
        "href": "http://graphviz-ewf.rhcloud.com:80/login",
        "rel": "login",
        "prompt": "User Login"
      }
    ]
  }
}

Graphviz is open source graph visualization software. The Graphviz API was designed to allow new users to register, add graph definitions and retrieve those definitions in various representations (pdf, jpg, png, gif). From the example API requests that are available in the Graphviz documentation, this looks like a well-conceived implementation of Collection+JSON. As authentication is required to use the Graphviz service I decided to continue my tour and found a Collection+JSON Browser. The browser functions much like the HAL browser except that the Hypermedia server is de-coupled from the client. The example Employee data is open and this means we can hit the endpoint directly from .. yep that’s right our friend the REST client.

Employee data in C+J representation
The above example shows a response from the Employee test data. Like HAL, Collection+JSON has a strong developer community.

Siren

Let’s complete our excursion with the newest design reviewed here. Kevin Swiber describes Siren as:

a hypermedia specification for representing entities

And like HAL and Collection+JSON, Siren has a registered media-type.

application/vnd.siren+json

To make our “hello-world” request to Siren we’ll use a Siren Browser developed by Wurl. Incidentally it’s great to see commercial API providers, such as Wurl and Graphviz embracing Hypermedia designs (this is the future :-)). Let’s point this Hypermedia client at a test Hypermedia server that Kevin has running on heroku. As this service is read only (404, “Cannot POST /users”) we cannot use it to make a “hello world” but the action for creating a user seems clear enough from the response to the initial GET.

GET http://siren-alps.herokuapp.com/
accept:application/vnd.siren+json

Here is the response. It has been trimmed down to the essential bit.

{
    "actions": [
        {
            "name": "user-add",
            "href": "http://siren-alps.herokuapp.com/users",
            "method": "POST",
            "fields": [
                {
                    "name": "user",
                    "type": "text"
                },
                {
                    "name": "email",
                    "type": "text"
                },
                {
                    "name": "password",
                    "type": "password"
                }
            ]
        }
    ]
}

Like the others, the Siren gang also hang out and share stuff.

Final thoughts

Something that all of the three Hypermedia designs have achieved is that at no point was it necessary to Read-The-Fscking-Manual. This is a Very Good Thing for Developer Experience. This is another clue for our understanding of Hypermedia design. Self-discovery! As said earlier, each design also has a high-level of abstraction. The flexibility provided by this abstraction does however make design selection seem rather arbitrary. Which of the three Hypermedia designs are right for me?!? I hope this will be the subject of my next excursion. Thanks.

API Design for large hierarchies

Introduction to the Problem

Some things that you may, or may not know about a Wolverine.

{
    "aliases": [
        "glutton",
        "skunk bear"
    ],
    "range": "620 km",
    "weight": "9-25 kg",
    "topSpeed": "48km/h",
    "lifestyle": "Solitary",
    "lifeSpan": "10-15 years",
    "conservationStatus": "Threatened",
    "habitat": "Mountainous regions and dense forest",
    "culture": "Michigan is known as the Wolverine state",
    "claimToFame": "In order to steal a kill a wolverine attacked a polar bear and clung to its throat until the bear suffocated."
}

Given that the the taxonomy of the living world contains around 8.7 million species. The challenges presented by this wolverine snippet are all about catalog and index management. The navigation of a large hierarchy is therefore something that API designers need to address when designing their interfaces. REST defines constraints that help us organise resources, but REST isn’t aligned to any particular data model. We therefore need to work out how best to manage hierarchical data ourselves. This post is based on experience from the design of a working system.

Limitations of the client

Rather than tackle the tree of life, I will use a tree of “things” to help model the API design. But firstly we need to understand two important limitations of our imaginary clients.

  1. The maximum payload our client can manage is 512 kilobytes, i.e. half-a-meg.
  2. The client is pretty dumb and will not be able to traverse the hierarchy without assistance.

Introducing a tree of things

The total number of things in our fictitious tree is 258. Things are nested down to a depth of three. Each thing contains exactly six other things. The following diagram shows a slice of the tree.

a diagram showing payload hierarchy

The things are named using the following convention.

A1		B10 .. B16		C100 .. C106
A2		B20 .. B26		C201 .. C206
... 
A6 		B60 .. B61 		C600 .. C601

In the tree of things the smallest payload that a node can return is 96 kilobytes. Doing the maths and the total size of the model works out at 4.03 megabytes (4,128 kb).

level kilobytes numOfItems
A 96 6
B 576 36
C 3,456 216
4,128 258

In the common scenario the client would access the API with the following request.

GET /things/A/B/C/204

And the server would respond as follows:

{
    "id": "C204",
    "items": [ 1, 2, 3, 4, 5, 6 ]
}

We are pretending that each item is 16 kilobytes and so we’re happy that the total 96 kilobytes is well within the required payload limit. This okay for the happy path but as-it-stands the design exposes a lot of loose endpoints. For example, what should the server do with the following request?

GET /things/

The server cannot respond with everything in A, plus all the B things together with their C descendants because the payload would be 4.03 mb, i.e. the full hierarchy. Perhaps a more reasonable response would be to remove the B and C descendants leaving just those in the range A1 to A6. Hmm, but now we’re starting to make assumptions about the request .. let’s play safe for now and just tell the client they asked for too much.

GET /things/ 				# 416 Requested Range Not Satisfiable

Using this approach I can complete my API design.

GET /things/A/B/C/204 		        # 200 Success 
GET /things/A 				# 416 Requested Range Not Satisfiable
GET /things/ 				# 416 Requested Range Not Satisfiable
GET /things/A/1/B/10		        # 200 Success
GET /things/A/3 			# 200 Success

At this stage the design is functional as it satisfies our minimum payload criteria. But it isn’t that easy to navigate. The top-level responses are blocked, and those responses that are returned are simply a flat list. It’s hard for my dumb client to know what to do with these. A response like this would be an improvement.

{
    "id": "A3",
    "items": [
        {
            "id": "B30",
            "items": [
                {
                    "id": "C300",
                    "items": [

	                    /* more items here */
                    ]
                },

                /* and more here too */
            ]
        }
    ]
}

Although this creates a payload problem (it weighs in at 688 kb) it shows promise because I can start to educate my client about the nature of the hierarchy.

Using depth to control the payload

To help the client get to know the tree of things without breaking the payload, I add the following parameters to my design.

GET /things/A/3/A/B

This meaning of the additional A/B parameter is to instruct the server to give me the descendants of B, as well as the list of A items that were discussed previously. Here’s the response.

{
    "id": "A3",
    "items": [
        {
            "id": ["B31", "B32", "B33", "B34", "B35", "B36"]
        }
    ]
}

Effectively I’ve filtered out C and thus got my payload down to 112 kilobytes. The client has a response that matched the request and thus enough information to start the descent into the hierarchy.

Using hypermedia to improve on the 416 method and help discovery

The new controls allow the client to control the depth of the nested response but there is still room for improvement. If the client initially goes for data that is out of the bounds of my payload limit, then the server must still return an error.

GET /things/A/3/A/B/C 				# 416 Requested Range Not Satisfiable

After receiving the 416 response they have to try again by trimming the depth back to A/B. But how can my dumb client figure this out from a 416 status code? This is where  HATEOS can help! As the server knows the payload limits it can construct compliant URLs and pass those onto the client. For example.

{
    "id": "A3",
    "links": {
        "next": "/things/A/3/A/B",
        "prev": null
    }
}

Using the links part of the response we can now return a 200 whenever the request has gone out-of-bounds. The links redirect the client towards the part of the hierarchy that can be reached from the current location. To achieve this the client has some simple logic to perform.

if (res.links) {
	callService(res.links);
} else if (res.items) {
	renderItems(res.items);
} else {
	// panic!
}

In summary, the features of the API design.

  • Allow the depth of the response to be specified in the request.
  • Return links rather than errors when the requested payload size is excessive.
  • The navigational links should be sensitive to the current location in the hierarchy.
  • The links communicate to the client the maximum depth that a resource can support when providing a response.

To finish off let’s briefly return to our wolverine. Assuming that we are able to discover the wolverine endpoint through navigation, we would end up with something like the following.

GET /species/vertebrates/carnivores/weasels/wolverine

{
    "id": "wolverine",
    "items": { /* snipped wolverine facts */ },
    "links": {
        "next": [
            "/species/vertebrates/carnivores/weasels/wolverine/luscos",
            "/species/vertebrates/carnivores/weasels/wolverine/gulo"
        ],
        "prev": "/species/vertebrates/carnivores/weasels"
    }
}

The wolverine item fits ours size requirement and so we get the payload. As it turns out there are two sub-species (American and European) we get some further navigation too. It would be fun to prove this out with a really big data set and see how well the model holds up. I hope this walk-through has illustrated some of the problems and solutions surrounding API design and large hierarchies.

Are WADLs yesterday’s news?

It has been four years since Sun’s Marc Hadley put forward his proposal for a Web Application Definition Language. Since the arrival of the WADL, we have seen the API Economy boom and an almost overwhelming swell towards REST and away from SOAP. Given that the WADL was intended to be the RESTful answer to SOAP’s WSDL you might have expected that the volume of WADLs would be rising in equal proportion. Somehow it hasn’t happened like that, so what’s going wrong?

The basic idea was that the WADL defined your REST interface in a way that machine generated code could consume it. After all, APIs are aimed at machines more so than humans. But a review of the clients that have actually committed code to the WADL specification suggests that outside of the Java community take-up hasn’t really happened, and even within Java it is fairly sporadic. To understand what’s going on here, let’s dig a little deeper into what a WADL currently offers.

UML diagram showing a WADL

From this UML summary of a WADL you should be able to see that a number of resources have been defined and attributed a URI, together with a method and HTTP verb. Within the resource definition are the details of the request/response cycle. For a full description try the Apigee WADL reference or the original submission to W3C .

Detractors of the WADL talk about true Hypermedia clients exploring the interface through the API responses. I think this is valid. But defining the contract between the client and the server is not the only purpose of a WADL. It can also be a general description of how an API is supposed to behave, even an API that hasn’t been built yet. Commissioning an API requires the establishment of a different kind of contract; a contract between a Business Owner and an API Designer. Using a WADL to express an API design maybe a side-effect of the original purpose, however this discussion is really more about using a WADL in the context of API design than it is about implementing a REST client.

Much of the original motivation for the WADL seems to have been around providing a simple and immediate alternative to the WSDL. The fact that it hasn’t been widely adopted suggests that the specification has gone too far the other way. Is it overly simplistic? If so, what is missing – what should a future WADL specification contain in order to overcome these limitations. Here is my WADL wishlist.

  • UML and Documentation
  • Inheritance
  • Interpolation
  • Managing Application State
  • Cross-reference
  • Non-functionals
  • JSON + node.js module

UML and Documentation
Most software design is done using UML. When it comes to applying the REST architectectural style there is surprisingly little around in the UML world. For example, the popular UML tool Enterprise Architect supports only WSDLs. This is awkward when attempting to visualise RESTful designs in an Enterprise context. An experimental attempt at bridging this gap is the eaWadl tool that imports a WADL into Enterprise Architect (disclaimer I wrote it).

In addition to using the WADL as part of the design process, another good use-case is documentation. For example, the Web Application Definition Language is used by Enunciate as a framework for human-readable content. The flexible <doc> tag really helps as it allows sample JSON or XML payloads to be embedded alongside the request and/or response. A common and effective ploy is to render the WADL documentation as HTML in response to an Accept Header of text/html (which a browser sends by default).

State
With a Home Document you don’t need a WADL to define the interface up-front, just grab the index and off-you-go. This is the logical extension of the Hypermedia approach. To explore this idea, let’s imagine that we have a REST interface to describe a game of chess. It should be straight-forward enough to imagine how each chess piece could be given a resource endpoint that would describe the various behaviours of each piece. For example a GET /pieces/king would provide a response that says move one square in any direction. Okay that’s fine. But what if we want to record a game of chess? This is a whole new tin of worms; that of managing the state of the game board. For example PUT /board/12/60/Qf2+. In it’s response the server should be helping the client to manage the application state by offering a link to /board/12/60. This is because only the server knows (with authority) that the game on board 12 is on the sixtieth move, that the White Queen has put the Black King in check and it is now Black’s turn.

In this scenario a WADL can be comfortably used to define the static reference data:  GET /pieces/king. But because the game state is (obviously) unknown at design time, a different approach is needed. The approach should involve making general statements about the Hypermedia syntax used by the API to maintain state. The request and response elements offered by the current WADL don’t constrain how the syntax should be defined. But I would go further and argue that it should. Elements such as <id>, <rel>, <class> and <href> have become so baked into Hypermedia designs that they deserve an exclusive place in our new generation WADL.

Cross-reference
A general issue with APIs is that of API discovery. How are you supposed to find out about the existence of an API in the first place? And if you are managing a group of related APIs, how does the discovery of one API help you find out about the other members of your API family. Google have tackled this problem through their discovery API. But what about everyone else? As the WADL already allows us to reference external Xml Schemas, why shouldn’t we be able to reference other WADLs?

<grammars>
<include wadl="http://example.com/xyz/doc.wadl"/>
</grammars>

With over 9,000 APIs currently registered on Programmable Web cross-referencing should help bring some organisation to the API space.

Inheritance
It’s generally polite to reply to a question in whatever language was used to ask the question. Consequently it’s annoying that each <request> and <response> declaration needs a separate <representation> tag. Although the WADL is hierarchical there is no sense of a child node that inherits from the parent. Wouldn’t it be easier to declare the base representation in the root <resources> tag, and use the child nodes to override any exceptions?

Non-functionals
Functionality is one thing, but if the security features of the API disallow access then it’s game over. If our new WADL has helped you discover an API then it seems logical that some declaration of the required authentication is made at the point at discovery. For example <authn> and <authz> tags might have values such as “oauth” or “x509” together with a <profile="developer.example.com/register"> statement for humans. Similarly a <status> parameter could broadcast downtime whenever an origin service becomes unavailable.

Error Handling
While you can get a long way with HTTP status codes many error responses are specific to a particular domain. A common strategy is therefore to return a <code>200</code> to show that the underlying HTTP communication was successful and then maintain a set of API specific error codes. The status attribute of the <response> element should be able to document the existence of the bespoke messages.

Interpolation
APIs are deployed in multiple environments. In reality the <resources> tag is likely to resolve to a number of hosts.

  • http://dev.example.com/api
  • https://example.com/api

Rather than relying on an external tool (such as Maven) it should be possible to abstract the hostname for use in multiple environments.

JSON + node.js module
With NodeJs currently the most second popular download on github, the march of JSON seems set to continue. WADLs are of course expressed using XML. Frameworks such as deployd illustrate the immediacy of using Node to define and create a REST service. With a little work, it’s possible to start this process from a WADL. With a bit more effort it would be possible to encapsulate the WADL import process using an Node module, similar to the work underway at RestDoc.

In summary, we should stick with the WADL not particularly because of it’s original intentation as a machine-readable definition of a service, but because the process of designing and developing APIs is lacking a reference language. The WADL just seems like a sensible place from which to embark.

Reading List

A fairly random collection of geeky things that I’m working my way through

Things I have read recently