An idealistic solution to the Spanish housing crisis

The central government of Spain should sponsor a programme to address the problem of incomplete building works. The role of the government should be to facilitate a programme that is implemented by each of the Spanish regions. The scale of the problem differs between regions, for example Andalucia has a large number of incomplete buildings littered along the Costa Del Sol. The central government should work with the central banks to provide a loan to the region that is proportionate to the reconstruction effort. Each region would then create a backlog of developments rated according to the cost of completion.

spanish building site
An abandoned building site in Spain

The region would then sponsor the formation of housing co-operatives. Each co-operative would draw on a mixture of unemployed people and those with experience in construction. A co-operative would apply to take ownership of a building site. The criteria for awarding a site to a co-operative would be based on the following:

  • The estimated capability of the co-operative to fulfill it’s promise
  • The amount of loan required to fund the re-construction
  • A loan repayment plan based on a mixture of rental earnings and sales from the salvaged properties
  • The destruction of one or more sites that have been rated as un-salvagable

During the re-construction and destruction period, co-operative members would draw on the loans to provide a social wage and funds for building materials. As habitable properties emerge from the projects, co-operative members can re-imburse themselves through either a period of rent-free occupation, or by selling on the open market and keeping a share of the profit. The remaining income would then be used to settle the initial loan to the central bank.

In this way, all the incomplete sites would either be converted to habitable dwellings or the land made suitable for re-use. A positive side-effect of the programme would be to reduce unemployment amongst young Spanish people.

It is possible that such plans exist already, but from my reading of (English-written) media it would appear that the hopes of the Spanish government rely entirely on foreign investors. The above solution maybe idealistic but it would offer the Spanish people autonomy, and control of their own destiny.

The Domain Name System modelled according to Archimate

a diagram explaining domain names
The Domain Name System modelled according to Archimate

The Archimate modelling language is a way of describing Enterprise and Solution architectures. In the interests of learning Archimate I’ve attempted to model the universally well-known Domain Name System in the diagram opposite. I’m not really expert in DNS either, but because DNS is a system that most Architects know *something* about, it seems like a good candidate for sharing some modelling knowledge.

The tool used to render the model was Archi. Exported as a CSV file for ease of transfer.

Letter to Obi-wan Kenobi

Dear Obi-wan,

Only you can save our community.

We are about 100k people living in a region blessed with sun and surrounded by the wonderful blue of the Mediterranean. Our community is not recognised by politicians and has no officially recognised borders. Our community has no name but we will call it La Gibranea for now. We live in the area from Europea Point in the South of Gibraltar to Santa Margarita in the North West. Including Campamento to the East but not as far as San Roque in the North. We are a single community. We live, love, work and die together but we are confused. Our community is divided and we need your leadership Obi-wan.

In La Gibranea we are even confused about what language to speak. For example, you will often hear “Spanglish”, an unwritten hybrid of Spanish and English. We are also confused about how to exchange goods for money; there are three currencies but where can we spend them? We are confused about everything: where to pay our taxes, buy our houses, send our children for education and our elderly for care. And we are very confused and frustrated about how to travel. Every morning we engage all conceivable forms of transport to carry ourselves across the hated border that divides us.

Floating ants

Like the senators in your Republic Obi-wan, the elected powers are too obsessed with themselves to see that we are adrift like the ants in this picture. Those elected are meant to help us but they cannot because each is strung to competing sovereigns. Obi-wan, you cannot understand our community when you are looking outside in. Viewed from the perspective of a nation state, La Gibranea does not make sense and this is the root of all our problems.

We need a new power to rule us and time to resolve our destiny. We need to create a new, transitional state with a mandate for a ten year rule. We need La Gibranea to be governed with a three-way sharing of power between Gibraltar, Spain and the UK. Each in equal measure.

During this time we need all the children born in La Gibranea to be granted dual nationality. We need a single currency. We need trade agreements to disincentivize the smugglers and proper recognition for our bi-lingualism. But most of all we need to abolish the border. We need to join the Schengen Agreement and establish freedom of movement through the introduction of an integrated transport system. These problems cannot be solved by the current leadership. Please Obi-wan, help bring balance to our community and recognise the state of La Gibranea.

Thank you.

Hello World in Hypermedia Style

After perhaps reading too much about Hypermedia (too much hype?) I started getting the itch to do something. This quick blog is a record of the steps I took towards a Hypermedia “hello world”. The main tool I used was a plug-in REST client from Mozilla but anything that lets you set headers and inspect responses will do the job.

For this exercise I choose three popular Hypermedia designs: HAL, Collection+JSON and Siren. Each design has a high-level of abstraction making it suitable for re-use and an active community of developers who are busily engaged with various implementations.


In the words of Mike Kelly, HAL’s creator :

HAL is a format you can use in your API that gives you a simple way of linking

See more of the HAL specification. HAL is available in either JSON or XML and has a registered media-type of


Mike has documented the HAL design by providing an interactive HAL Browser. This is the most immediate way to say “hello world” in Hypermedia style. Firstly you use the HAL browser to create an account (hint: see the hints :-)). Next you should navigate to /users/:account and then use the NON-GET button to get into the following dialogue box.

Saying it
Saying Hello World

It’s okay if you type something other than “hello world” *yawn* but changing anything else might break something .. To verify your creation navigate to the latest posts and as-if-by-magic your entry will appear (fingers x).

The HAL browser is great way to quickly get a feel for what HAL is all about. Armed with this knowledge I wanted to take a step towards a HAL client that could be controlled independently, and ultimately deployed in another context. The HAL browser seemed tightly coupled to the server, so rather than unpick it, I dug around and found another server running on the HAL builder site. I made the following request using my trusty REST client.

Accept: application/hal+json

And in case you weren’t paying attention earlier, here’s a picture.


The HAL builder dutifully obeyed my request (yes, the Accept header is necessary) and gave me the following response.

    "_links": {
        "self": {
            "href": "/restbucks/api"
        "orders": {
            "href": "/restbucks/api/orders"
    "name": "RestBucks Store",
    "project": "restbucks",
    "version": "0.0.1"

Now if I was a robot, I would crack-on and apply some automation to the above. Here are our first clues about Hypermedia. Custom media-types and machine-based connections! There’s way more discussion about using HAL in the  forum.


Mike Amundsen is the author of Collection+JSON and says it is

a JSON-based read/write hypermedia-type designed to support management and querying of simple collections

And just like HAL, Collection+JSON has its very own registered media-type. To get a test response from Collection+JSON we’ll return to our REST client. As before set the client up for the target service and specify the media-type.

Accept: application/vnd.collection+json

This Hypermedia server responds with the following.

  "collection": {
    "version": "1.0",
    "href": "",
    "links": [
        "href": "",
        "rel": "home",
        "prompt": "Home API"
        "href": "",
        "rel": "graphs",
        "prompt": "Home Graph"
        "href": "",
        "rel": "register",
        "prompt": "User Register"
        "href": "",
        "rel": "login",
        "prompt": "User Login"

Graphviz is open source graph visualization software. The Graphviz API was designed to allow new users to register, add graph definitions and retrieve those definitions in various representations (pdf, jpg, png, gif). From the example API requests that are available in the Graphviz documentation, this looks like a well-conceived implementation of Collection+JSON. As authentication is required to use the Graphviz service I decided to continue my tour and found a Collection+JSON Browser. The browser functions much like the HAL browser except that the Hypermedia server is de-coupled from the client. The example Employee data is open and this means we can hit the endpoint directly from .. yep that’s right our friend the REST client.

Employee data in C+J representation
The above example shows a response from the Employee test data. Like HAL, Collection+JSON has a strong developer community.


Let’s complete our excursion with the newest design reviewed here. Kevin Swiber describes Siren as:

a hypermedia specification for representing entities

And like HAL and Collection+JSON, Siren has a registered media-type.


To make our “hello-world” request to Siren we’ll use a Siren Browser developed by Wurl. Incidentally it’s great to see commercial API providers, such as Wurl and Graphviz embracing Hypermedia designs (this is the future :-)). Let’s point this Hypermedia client at a test Hypermedia server that Kevin has running on heroku. As this service is read only (404, “Cannot POST /users”) we cannot use it to make a “hello world” but the action for creating a user seems clear enough from the response to the initial GET.


Here is the response. It has been trimmed down to the essential bit.

    "actions": [
            "name": "user-add",
            "href": "",
            "method": "POST",
            "fields": [
                    "name": "user",
                    "type": "text"
                    "name": "email",
                    "type": "text"
                    "name": "password",
                    "type": "password"

Like the others, the Siren gang also hang out and share stuff.

Final thoughts

Something that all of the three Hypermedia designs have achieved is that at no point was it necessary to Read-The-Fscking-Manual. This is a Very Good Thing for Developer Experience. This is another clue for our understanding of Hypermedia design. Self-discovery! As said earlier, each design also has a high-level of abstraction. The flexibility provided by this abstraction does however make design selection seem rather arbitrary. Which of the three Hypermedia designs are right for me?!? I hope this will be the subject of my next excursion. Thanks.

API Design for large hierarchies

Introduction to the Problem

Some things that you may, or may not know about a Wolverine.

    "aliases": [
        "skunk bear"
    "range": "620 km",
    "weight": "9-25 kg",
    "topSpeed": "48km/h",
    "lifestyle": "Solitary",
    "lifeSpan": "10-15 years",
    "conservationStatus": "Threatened",
    "habitat": "Mountainous regions and dense forest",
    "culture": "Michigan is known as the Wolverine state",
    "claimToFame": "In order to steal a kill a wolverine attacked a polar bear and clung to its throat until the bear suffocated."

Given that the the taxonomy of the living world contains around 8.7 million species. The challenges presented by this wolverine snippet are all about catalog and index management. The navigation of a large hierarchy is therefore something that API designers need to address when designing their interfaces. REST defines constraints that help us organise resources, but REST isn’t aligned to any particular data model. We therefore need to work out how best to manage hierarchical data ourselves. This post is based on experience from the design of a working system.

Limitations of the client

Rather than tackle the tree of life, I will use a tree of “things” to help model the API design. But firstly we need to understand two important limitations of our imaginary clients.

  1. The maximum payload our client can manage is 512 kilobytes, i.e. half-a-meg.
  2. The client is pretty dumb and will not be able to traverse the hierarchy without assistance.

Introducing a tree of things

The total number of things in our fictitious tree is 258. Things are nested down to a depth of three. Each thing contains exactly six other things. The following diagram shows a slice of the tree.

a diagram showing payload hierarchy

The things are named using the following convention.

A1		B10 .. B16		C100 .. C106
A2		B20 .. B26		C201 .. C206
A6 		B60 .. B61 		C600 .. C601

In the tree of things the smallest payload that a node can return is 96 kilobytes. Doing the maths and the total size of the model works out at 4.03 megabytes (4,128 kb).

level kilobytes numOfItems
A 96 6
B 576 36
C 3,456 216
4,128 258

In the common scenario the client would access the API with the following request.

GET /things/A/B/C/204

And the server would respond as follows:

    "id": "C204",
    "items": [ 1, 2, 3, 4, 5, 6 ]

We are pretending that each item is 16 kilobytes and so we’re happy that the total 96 kilobytes is well within the required payload limit. This okay for the happy path but as-it-stands the design exposes a lot of loose endpoints. For example, what should the server do with the following request?

GET /things/

The server cannot respond with everything in A, plus all the B things together with their C descendants because the payload would be 4.03 mb, i.e. the full hierarchy. Perhaps a more reasonable response would be to remove the B and C descendants leaving just those in the range A1 to A6. Hmm, but now we’re starting to make assumptions about the request .. let’s play safe for now and just tell the client they asked for too much.

GET /things/ 				# 416 Requested Range Not Satisfiable

Using this approach I can complete my API design.

GET /things/A/B/C/204 		        # 200 Success 
GET /things/A 				# 416 Requested Range Not Satisfiable
GET /things/ 				# 416 Requested Range Not Satisfiable
GET /things/A/1/B/10		        # 200 Success
GET /things/A/3 			# 200 Success

At this stage the design is functional as it satisfies our minimum payload criteria. But it isn’t that easy to navigate. The top-level responses are blocked, and those responses that are returned are simply a flat list. It’s hard for my dumb client to know what to do with these. A response like this would be an improvement.

    "id": "A3",
    "items": [
            "id": "B30",
            "items": [
                    "id": "C300",
                    "items": [

	                    /* more items here */

                /* and more here too */

Although this creates a payload problem (it weighs in at 688 kb) it shows promise because I can start to educate my client about the nature of the hierarchy.

Using depth to control the payload

To help the client get to know the tree of things without breaking the payload, I add the following parameters to my design.

GET /things/A/3/A/B

This meaning of the additional A/B parameter is to instruct the server to give me the descendants of B, as well as the list of A items that were discussed previously. Here’s the response.

    "id": "A3",
    "items": [
            "id": ["B31", "B32", "B33", "B34", "B35", "B36"]

Effectively I’ve filtered out C and thus got my payload down to 112 kilobytes. The client has a response that matched the request and thus enough information to start the descent into the hierarchy.

Using hypermedia to improve on the 416 method and help discovery

The new controls allow the client to control the depth of the nested response but there is still room for improvement. If the client initially goes for data that is out of the bounds of my payload limit, then the server must still return an error.

GET /things/A/3/A/B/C 				# 416 Requested Range Not Satisfiable

After receiving the 416 response they have to try again by trimming the depth back to A/B. But how can my dumb client figure this out from a 416 status code? This is where  HATEOS can help! As the server knows the payload limits it can construct compliant URLs and pass those onto the client. For example.

    "id": "A3",
    "links": {
        "next": "/things/A/3/A/B",
        "prev": null

Using the links part of the response we can now return a 200 whenever the request has gone out-of-bounds. The links redirect the client towards the part of the hierarchy that can be reached from the current location. To achieve this the client has some simple logic to perform.

if (res.links) {
} else if (res.items) {
} else {
	// panic!

In summary, the features of the API design.

  • Allow the depth of the response to be specified in the request.
  • Return links rather than errors when the requested payload size is excessive.
  • The navigational links should be sensitive to the current location in the hierarchy.
  • The links communicate to the client the maximum depth that a resource can support when providing a response.

To finish off let’s briefly return to our wolverine. Assuming that we are able to discover the wolverine endpoint through navigation, we would end up with something like the following.

GET /species/vertebrates/carnivores/weasels/wolverine

    "id": "wolverine",
    "items": { /* snipped wolverine facts */ },
    "links": {
        "next": [
        "prev": "/species/vertebrates/carnivores/weasels"

The wolverine item fits ours size requirement and so we get the payload. As it turns out there are two sub-species (American and European) we get some further navigation too. It would be fun to prove this out with a really big data set and see how well the model holds up. I hope this walk-through has illustrated some of the problems and solutions surrounding API design and large hierarchies.

Common URL Pattern

When architecting with domains, services and URLs this pattern seems to come round a lot, but I’ve never heard of it being given a formal name or description:

protocol. service. env. example. com / product

I will call it the Common URL Pattern. In a real-world scenario it would resolve to something like: Here is the same pattern again but this time expressed by throwing some UML around.

UML diagram showing each part embedded inside it's parent
Common Url Pattern

And finally by way of explanation, some comments on the parts.

protocol Underlying transport layer such as https, smtp
service Shared across a technology stack: www, mail, api, db
environment Instances of the stack with functional variation, e.g. test, dev
domain Identifier in the Internet namespace: a.b.c
product A website or application that handles end-user interaction, e.g. /blah

It seems like a fairly adaptable model that can be applied to a large number of websites and web-based applications.

Are WADLs yesterday’s news?

It has been four years since Sun’s Marc Hadley put forward his proposal for a Web Application Definition Language. Since the arrival of the WADL, we have seen the API Economy boom and an almost overwhelming swell towards REST and away from SOAP. Given that the WADL was intended to be the RESTful answer to SOAP’s WSDL you might have expected that the volume of WADLs would be rising in equal proportion. Somehow it hasn’t happened like that, so what’s going wrong?

The basic idea was that the WADL defined your REST interface in a way that machine generated code could consume it. After all, APIs are aimed at machines more so than humans. But a review of the clients that have actually committed code to the WADL specification suggests that outside of the Java community take-up hasn’t really happened, and even within Java it is fairly sporadic. To understand what’s going on here, let’s dig a little deeper into what a WADL currently offers.

UML diagram showing a WADL

From this UML summary of a WADL you should be able to see that a number of resources have been defined and attributed a URI, together with a method and HTTP verb. Within the resource definition are the details of the request/response cycle. For a full description try the Apigee WADL reference or the original submission to W3C .

Detractors of the WADL talk about true Hypermedia clients exploring the interface through the API responses. I think this is valid. But defining the contract between the client and the server is not the only purpose of a WADL. It can also be a general description of how an API is supposed to behave, even an API that hasn’t been built yet. Commissioning an API requires the establishment of a different kind of contract; a contract between a Business Owner and an API Designer. Using a WADL to express an API design maybe a side-effect of the original purpose, however this discussion is really more about using a WADL in the context of API design than it is about implementing a REST client.

Much of the original motivation for the WADL seems to have been around providing a simple and immediate alternative to the WSDL. The fact that it hasn’t been widely adopted suggests that the specification has gone too far the other way. Is it overly simplistic? If so, what is missing – what should a future WADL specification contain in order to overcome these limitations. Here is my WADL wishlist.

  • UML and Documentation
  • Inheritance
  • Interpolation
  • Managing Application State
  • Cross-reference
  • Non-functionals
  • JSON + node.js module

UML and Documentation
Most software design is done using UML. When it comes to applying the REST architectectural style there is surprisingly little around in the UML world. For example, the popular UML tool Enterprise Architect supports only WSDLs. This is awkward when attempting to visualise RESTful designs in an Enterprise context. An experimental attempt at bridging this gap is the eaWadl tool that imports a WADL into Enterprise Architect (disclaimer I wrote it).

In addition to using the WADL as part of the design process, another good use-case is documentation. For example, the Web Application Definition Language is used by Enunciate as a framework for human-readable content. The flexible <doc> tag really helps as it allows sample JSON or XML payloads to be embedded alongside the request and/or response. A common and effective ploy is to render the WADL documentation as HTML in response to an Accept Header of text/html (which a browser sends by default).

With a Home Document you don’t need a WADL to define the interface up-front, just grab the index and off-you-go. This is the logical extension of the Hypermedia approach. To explore this idea, let’s imagine that we have a REST interface to describe a game of chess. It should be straight-forward enough to imagine how each chess piece could be given a resource endpoint that would describe the various behaviours of each piece. For example a GET /pieces/king would provide a response that says move one square in any direction. Okay that’s fine. But what if we want to record a game of chess? This is a whole new tin of worms; that of managing the state of the game board. For example PUT /board/12/60/Qf2+. In it’s response the server should be helping the client to manage the application state by offering a link to /board/12/60. This is because only the server knows (with authority) that the game on board 12 is on the sixtieth move, that the White Queen has put the Black King in check and it is now Black’s turn.

In this scenario a WADL can be comfortably used to define the static reference data:  GET /pieces/king. But because the game state is (obviously) unknown at design time, a different approach is needed. The approach should involve making general statements about the Hypermedia syntax used by the API to maintain state. The request and response elements offered by the current WADL don’t constrain how the syntax should be defined. But I would go further and argue that it should. Elements such as <id>, <rel>, <class> and <href> have become so baked into Hypermedia designs that they deserve an exclusive place in our new generation WADL.

A general issue with APIs is that of API discovery. How are you supposed to find out about the existence of an API in the first place? And if you are managing a group of related APIs, how does the discovery of one API help you find out about the other members of your API family. Google have tackled this problem through their discovery API. But what about everyone else? As the WADL already allows us to reference external Xml Schemas, why shouldn’t we be able to reference other WADLs?

<include wadl=""/>

With over 9,000 APIs currently registered on Programmable Web cross-referencing should help bring some organisation to the API space.

It’s generally polite to reply to a question in whatever language was used to ask the question. Consequently it’s annoying that each <request> and <response> declaration needs a separate <representation> tag. Although the WADL is hierarchical there is no sense of a child node that inherits from the parent. Wouldn’t it be easier to declare the base representation in the root <resources> tag, and use the child nodes to override any exceptions?

Functionality is one thing, but if the security features of the API disallow access then it’s game over. If our new WADL has helped you discover an API then it seems logical that some declaration of the required authentication is made at the point at discovery. For example <authn> and <authz> tags might have values such as “oauth” or “x509” together with a <profile=""> statement for humans. Similarly a <status> parameter could broadcast downtime whenever an origin service becomes unavailable.

Error Handling
While you can get a long way with HTTP status codes many error responses are specific to a particular domain. A common strategy is therefore to return a <code>200</code> to show that the underlying HTTP communication was successful and then maintain a set of API specific error codes. The status attribute of the <response> element should be able to document the existence of the bespoke messages.

APIs are deployed in multiple environments. In reality the <resources> tag is likely to resolve to a number of hosts.


Rather than relying on an external tool (such as Maven) it should be possible to abstract the hostname for use in multiple environments.

JSON + node.js module
With NodeJs currently the most second popular download on github, the march of JSON seems set to continue. WADLs are of course expressed using XML. Frameworks such as deployd illustrate the immediacy of using Node to define and create a REST service. With a little work, it’s possible to start this process from a WADL. With a bit more effort it would be possible to encapsulate the WADL import process using an Node module, similar to the work underway at RestDoc.

In summary, we should stick with the WADL not particularly because of it’s original intentation as a machine-readable definition of a service, but because the process of designing and developing APIs is lacking a reference language. The WADL just seems like a sensible place from which to embark.

The Street Performers of El Rastro

Had a great day in Madrid where I was amazed and entertained by some amazing street performers.

beautiful boleros

energy and ancestry from simplicity

keeping the old glasses humming

when the kids are happy ..

trad jazzers










skin deep

mud statue





API Predictions

This is a fantasy view of the future inspired by talking with people at APIdays. It’s a personal view and best read as science-ficiton, but you never know, some of it may come true!


Predictions in this section seem fairly likely to happen if not happening already

  1. A few branded, skyscraper APIs will continue to be dominant and used by the vast majority of apps. The number of skyscapers will represent a tiny minority of the total APIs publically available.
  2. The dx (developer experience) movement will get into full swing and impact the API design space.
  3. Hypermedia will formalise itself as a series of Architectural Design Patterns. See [1], [2] and [3] for examples.
  4. HTML5 and websockets will spawn a new generation of real-time APIs


These predictions are not happening yet, but reasonably likely to come into place

  1. Apigee will have been acquired.
  2. Twitter will win back their developer community.
  3. NodeJs applications such as deployd [4] will continue to drive down availability of XML representations. XML may disappear from the majority of public APIs, but will continue to live-on in bespoke partner and private APIs.
  4. ProgrammableWeb will become obsolete. Instead there will be a new generation of API discovery tools [5]
  5. There will be a significant increase in the number of stand-alone API providers such as Twilio. (Twilio is a stand-alone API provider because it doesn’t have a core product in the way that Google, Facebook and Twitter do.)
  6. An open-source and independent solution for API traffic management will become available, perhaps based on [6]. API clients will become more adept at self-managing load.
  7. APIs will provide text/html as their default content-type (because humans need to understand them before the machines can get started).
  8. There will be a single go-to place for API developers to social network (and it won’t be Facebook).
  9. API security will change radically in response to the general availability of personally available hardware/mobile tokenisers.
  10. Commercial monitoring tools will get-in on the API act. Their solutions will provide views of how a single request passes through the technology stack. The apiKey will provide the glue.


Predictions that will probably never happen, perhaps this is a wish-list ..

  1. W3C will publish an API standard that is largely driven by Hypermedia and the requirements of the API clients. Kin Lane [7] will be involved!
  2. Semantic Web will align with a new generation of media-types that arise from the rising popularity of the Hypermedia style. A standard representation for RDF and JSON will emerge, driven by a skyscaper API provider and (eventually) blessed by W3C.
  3. A famous legal battle will project the issue of API provenance into the media. Digital signature solutions will evolve and adapt themselves to the API economy.
  4. Single Page Applications (SPAs) will be the accepted alternative to traditional page-based web sites. SPAs will use registered media-types.
  5. The Great-Twitter-Betrayal will give rise to a credit-rating style system. This system will make promises to Venture Capitalists and make them feel better about manging risk in the API economy. Everyone else will ignore it.
  6. Software tools for API management (see [8] and [9]) are provided out-of-the-box by the majority of cloud vendors.
  7. Delays in the transition from HTTP 1.1 to HTTP 2.0 will engender the adoption of a new protocol that will be optimised for API traffic and messaging.
  8. A single supplier will capture the market for API developer portals