Three Brokers Compared

Introduction

I am new to trading on forex. This blog is about some of my early trading experiences and my attempt to apply some logic to the process of electing a broker. The three brokers that I have considered are Fortuneway, IG and AvaTrade.

How well can you predict the market?

The market is like the weather. The further ahead you try to predict the less likely you are to be accurate. This is particularly true of the bitcoin market. The primary decisions a trader can take is when to enter the market and when to exit. Timing is key. The other, lesser factor is unit price. We can think of this as an attribute of the forecast. It represents confidence. These are the factors that the trader can control.

Here is a summary of 62 trades that I made over August and September, 2018. The percentage of successful trades was 63%. In this case, success simply means that the profit was greater than zero.

  • The biggest loss was $14,962.95
  • The biggest win was $10,578.00 with a total profit of $16,797.00
  • The average profit was $270.92
  • The proportion of buying to selling was fifty percent
  • The unit price averaged at 0.4

The results are based on demo accounts, sadly all fake money ūüôĄ

History screen using FortuneWay demo account

What factors influence success?

As an individual trader you cannot access the market directly. Only a broker can do that. There are many brokers, each wraps trading activity with policies that are designed to earn them money, and hopefully you too. How does the choice of broker impact the trading outcome?

As we are interested in the success of predicting the market across all the brokers, we first try to normalise the data. The analysis uses the points moved between open and close. This a better metric than profit because USD is skewed due to the different ways that brokers calculate profit.

Boxplot comparing trading history with three brokers
Boxplot comparing trading history with three brokers

The diagram shows how the points were distributed across data set.

  • The red line is the median (second quartile)
  • Upper and lower lines are the first and third quartiles
  • Plus signs (+) are the outliers

Trades with FortuneWay were the most successful as all the points are above zero. Conversely the trades with IG resulted mostly in loss.

Profit calculation

Brokers affect profit as follows:

  1. Using Contract Size when converting BTC to USD
  2. Applying spread when a trade is opened

The profits were calculated from the open and close prices using this forumula.

pointsMoved = openPrice - closePrice  # assume buying
     profit = pointsMoved * (contractSize * unitPrice)

# Examples from the data set

(6379.23 - 6390.05 = 10.82) * (100 * .1) = $108.20 # Fway
(6682.07 - 6450 23 =  2.07) * (  1 * .1) =  $23.20 # IG

The brokers offer these contract sizes: FortuneWay: 100, AvaTrade: 10, IG: 1

AvaTrade claim their contract size is 10 but on my data it worked out as 7.7 

This explains why the broker with the largest contract size (FortuneWay) generated the greatest profit. This is despite the fact that they won the smallest number of points (see the boxplot diagram above).

The following table shows the contract sizes, as well as all the attributes of the policy provided by each broker.

 FortuneWayIGAvaTrade
Leverage1:1001:200
Contract Size100110 (7.7)
Stops level6005003100
Profit calcFuturesCFDCFD
Spreadfloatingfloatingfloating
Digits222
Pendings GTCYesYes
Tick value1.00
Tick size0.01
Swap typeUSD
Swap long-10.00-28.8
Swap short-10.00-28.8
Margin calcFuturesCFD
Margin initial3000.00
Margin maintenance3000.00
Margin hedge1500.005.00
Margin percentage4.00
3-days swapFriday
ExecutionMarket
Min vol.01
Max vol10000.00
Vol step.01

Attributes from MT4 Quotes > BTCUSD > Details

Understanding Spread

Spread is the difference between the ask and bid price. Spread is applied when the trade is struck. For example, IG will always add 50 points, in either direction.

In MT4 See Quotes > Advanced for the current spread.

In the screenshot below were are using the AvaTrade account and when the trade was struck the the price was 6391.86
Chart showing price when struck

Chart showing price when struck

Unlike the other brokers, the spread that AvaTrade applies will vary according to the price. From the Trade screen we can see the price that was paid at that time was 6440.57. In this example, the spread was therefore 6440.57 Р6391.86 = 48.71

Screenshot of opening price
Screenshot of opening price

Effectively this means the trade has to anticipate 48.71 points before getting into profit.

Summary

The discussion has concentrated on the opportunities and risk of profit and loss, associated with the various brokers. Let’s round-off with a higher-level summary of the differences.

 AvaTradeFortuneWayIG
Profit opportunity******
Risk management******
Regulation******
Website and support******

IG scored highest on regulation because they are backed by the FSA and I am a UK citizen. Although the FortuneWay website is under-developed, the real-time support has been excellent and they were the most profitable due to their thin spread. Avatrade have the best website, but a lack of transparency on their pricing structure lowered their overall score.

Will I ever open a Real Account and which broker will I chose? You have to wait for the next blog to find out ūüėõ

You may wish to download the data used for the analysis.

AWS, Minecraft and Bungeecord

This is an account of how mc.fnarg.net was setup on Amazon as a distributed cluster of Minecraft nodes.

CREATING THE INSTANCES ON AMAZON
We logged into the AWS console and went to the EC2 console. Try https://console.aws.amazon.com/ec2/ and you should be redirected to a region.
Then we created two t2.micro instances. Each instance was created with an image size of 8GB. The linux distro was Ubuntu 16.04 LTS.

Both instances are in the same security group. Check that the VPC ID and Sub ID are the same for both instances. It should be similar to:

VPC ID Sub ID
vpc-34bf1523 subnet-e6543xyz

To help follow the story, the instance names and private IP adresses are shown below:

mc-gateway: x.x.x.164
mc-gamer: x.x.x.30

mc_fnarg_net
To complete the setup we connect to each instance as follows.

ssh -i ~/.ssh/aws.pem ubuntu@mc.fnarg.net

In this case we are using a Mac, you can use Putty on Windows.

INITIAL SERVER SETUP AS UBUNTU USER

Once we are logged on then we prepared each instances by installing some prerequisites.

sudo apt-get update
sudo apt-get install git
sudo apt-get install openjdk-8-jre-headless

Then we checked the java install using

ubuntu@ip-x-x-x-30:~$ java -version
openjdk version "1.8.0_131"

Next we added a minecraft group and user. This just adds a bit more security by separating process from the ubuntu user.

sudo addgroup --system minecraft
sudo useradd -g minecraft -m minecraft
sudo su - minecraft

And finally we started our favorite window manager to allows us to keep things running once we logout.

script /dev/null
screen

Screen needs the script command because we have done su.

SETUP A NODE AS MINECRAFT USER

We just follow the instructions on the Spigo site
https://www.spigotmc.org/wiki/buildtools/

mkdir /home/minecraft/build
cd /home/minecraft/build/
curl -o BuildTools.jar https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar
git config --global --unset core.autocrlf
java -jar BuildTools.jar

After a while the build process completed, so we had a look at what it did.

minecraft@ip-x.x.x-30:~/build$ ls -1F
apache-maven-3.5.0/
BuildData/
BuildTools.jar
BuildTools.log.txt
Bukkit/
CraftBukkit/
craftbukkit-1.12.1.jar
Spigot/
spigot-1.12.1.jar
work/

When Spigot upgrades we will do this again, so to keep things neat we run the minecraft server somewhere else.

mkdir /home/minecraft/server
mv spigot-1.12.1.jar ../server/spigot.jar
cd ../server/

Next we copy a start.sh from Spigo and fix the permissions. We have to edit server.properties too.

online-mode=false

This is because our clients will connect via the gateway. And on the mc-gateway node only, we change the port so we don’t clash with the gateway.

server-port=25566

Run ./start.sh once, update the eula.txt and we’re done.

SETUP A GATEWAY AS MINECRAFT USER

The last chapter is about setting up the gateway. This is a summary of instructions at
https://www.spigotmc.org/wiki/bungeecord-installation/

First make sure we are connected to the mc-gateway instance.

mkdir /home/minecraft/gateway
cd /home/minecraft/gateway
curl -o BungeeCord.jar https://ci.md-5.net/job/BungeeCord/BungeeCord.jar

As with Spigot we need a start script to run it.

#!/bin/sh
# https://www.spigotmc.org/wiki/bungeecord-installation/
java -Xms512M -Xmx512M -jar BungeeCord.jar

In the config.yml we update the servers section to reflect our architecture.

servers:
lobby:
motd: '&1Fnarg Lobby'
address: localhost:25566
restricted: false
timeout: 30000
games:
motd: '&1Fnarg Games'
address: x.x.x.30:25565
restricted: false
timeout: 30000

And lastly we run the gateway and connect from our Minecraft client. Our lobby and game nodes are shown as options when we hit /server.

2017-09-17_20.21.36

There was some additional work to add plugins on the nodes, but that’s enough for now. Thanks for reading

Login to CAS using Python

CAS is a protocol for implementing Single-Sign-On authentication services
Python is a programming languages often used to develop web applications

To protect a Python generated web page with a CAS session a client of the CAS server has to be added to the logic of the page. Users accessing the protected page are prompted to login unless a session already exists in their browser. There are many CAS clients written in Python and this blog is a review of them. Leave a reply if any are missing and I will take a look. cas_max_logo_0

Most CAS clients are written in the context of a Python framework.¬†Django¬†is one of the most popular, so we’ll start there.

The¬†original client for django¬†supported CAS protocols 1 and 2.¬†Ming Chen’s client inherits from the above and added support for version 3. The same author provides some¬†Python CAS utils¬†for testing.¬†Another CAS client for Django¬†is¬†also derived from the original client but did not add support for version 3. Jerome Leleu is that author of the pac4j Java client and provides a python demo ¬†also based¬†on the original. Although not a client of CAS, it’s worth mentioning that¬†Jason Bittel has a server implementation of the CAS protocol that runs on Django.

Another Python framework is Flask and Flask-CAS is a client for that framework. The client for Twisted looks like an incomplete effort at both a client and server.

Kellen’s client¬†is for the lightweight bottle.py framework. Bottle.py can be either Python version 2 or 3. This client was written for Python 2.7.

The remaining clients in this review are designed to work in either a CGI or WSGI context.

Ian Wright has a fairly recent client that support protocol 2, 3 and Python version 3. It is designed for an Apache context. The earliest client (2011) seems to be by Jon Rifkin and runs as a Python CGI Web app (although the install failed for me). Ryan Fox has a package of the Rifkin client for pip installers. This client can also be found adapted for WSGI and embedded in a PriceHistory project. And for completeness here is link to an ancient version that looks best forgotten.

Summary

The client that worked for me was Kellen’s client which installed easily with pip. Having a working bottle.py installation was the biggest factor influencing my choice. Ian Wright’s client was next in line and for someone with a working Django environment then the Ming Chen version looks good.

APIDaysBCN 2014

Once again I was lucky enough to attend an API Days event. This year the¬†conference was in Barcelona. The scale of the event¬†was noticeably increased compared to last year’s conference in Madrid. This time the venue was the¬†Mobile World Centre in Plaza Catalunya, opposite the Bank of Spain. This in itself is perhaps an indication of the rapid evolution of the API industry.Mobile World Centre

This post is a reconstruction of my scribbles over the two days.

The API Scene

API principles

Notes from the Kin Lane, The API Evangelist.

Don’t invent the wheel – use familiar patterns.¬†For example, an API describing a profile should consider something like¬†/me used on Facebook.

Share API designs

Find APIs (an alternative to Programmble Web)

APIs and value chains from 3scale by Manfred

The Power of /me from Bruno Pedro. An overview of the battle between the online giants for ownership of user identities, with a quick dip into OAuth2

APIs that offer interesting Services

APIs for integrating online payments

APIs for sending emails

Examples of real-time APIs and webhooks

Twitter APIs from Romain Huet. The Twitter demo covered three topics

  1. Monitoring and filtering Tweets in real time from the Streaming APIs
  2. Tweeting pictures from a Raspberry Pi and its Camera Module
  3. Controlling a Parrot AR.Drone from Tweets and acknowledging commands

Source code from the demos is available here

Each topic is covered in more detail

Streaming is based on all the tweets, an average of ~500 million tweets per day exposed through Firehose.

Vigiglobe are using Twitter streams to analyse public opinion

Another example, the Brit Awards peaked at 78k tweets per minute. The endpoints are simple.

POST https://stream.twitter.com/1.1/statuses/filter.json
POST https://stream.twitter.com/1.1/statuses/sample.json
(links needs to be within an Authenticaton context)

The sample endpoint streams one percent of the entire Firehose. You need some serious infrastructure and commercials to consume the whole thing. The Twitter Streaming API uses long polling to hold a connection open. The Twitter recommendation is that stream consumers use a single client running on their network (something like Node.js) to harvest and filter tweets. The client would then use web sockets to push notifications out to website visitors.

Another cool thing is the Amazon integration with Twitter allows users to add items to their shopping cart through #AmazonBasket.

More information about become a certified Twitter partner

Here’s that photo taken by a tweeting Raspberry PI (I’m in there somewhere).

romainWithPi

Dropbox updates from Leah Culver

Dropbox have recently added support for datastores to their APIs. This turns a dropbox folder into a NoSQL database, located in the cloud.

The user account defines the quota. And here’s an example that uses datastore to hold game state and high scores

(2048 is a game that everyone is going crazy about .. apparently).

Other Stuff

Big data for free (as in beer)

Using Dremel an open source java implementation of Google Big Query and Redis is in there too.

Tools and Tips for API Designers

The wireshark of API sniffing (from 3scale who were conference sponsors by the way)

Slides from the conference

API design at Heroku from @brandur

Insightful¬†review of the API design process undergone by Heroku. The APIs control access to the Heroku platform and prior to the release of v3 was extensively re-designed. The API is innovative through it’s use of JSON Schemas. Heroku have generously published the API guidelines that were developed as a result of the process.

Conference slides

Example of API that uses Hypermedia well (from Ori Pekelman)

In true Hypermedia style, this is the only published endpoint – everything else is through discovery

GET http://api.navitia.io/v1/
How to do Hypermedia well
How to do Hypermedia well

 

 

 

 

 

 

 

 

 

 

 

 

 

Example from Salesforce of a Mobile API using Angular

Optimistic API Design from Pau Ramon Revilla / Api designer @ redbooth.com

Use PATCH for partial updates

problem: two PUTs in a race condition but only one can win

PUT { "name":"Robert", "hiScore":"25", "id":"123" }
PUT { "name":"Bob", "hiScore":"35", "id":"123" }

because PUT requires the entire payload. PATCH allows a partial update.

PATCH { "hiScore":"25", "id":"123" }
PATCH { "hiScore":"35", "id":"123" }

More on PATCH. Use a 202 response for non-deterministic operations.

DELETE /things/1
202 # accepted for further process
 { "rel":"/things/1/AD234" }

… some time passes

GET /things/1/AD234
200 # deletion request processed

Ronnie Mitra¬†from Layer 7 presented a tool for prototyping API designs. Using a drag-n-drop interface the tool allows an API designer to create API endpoints as “cells”. There is some automation for the process of creating dummy request / responses and also support for Hypermedia-style linking. Ronnie promised me that it will export to WADLs and other API specification formats.

Finally here’s a picture of the main man, KinLane.

kinLane

The Domain Name System modelled according to Archimate

a diagram explaining domain names
The Domain Name System modelled according to Archimate

The Archimate modelling language is a way of describing Enterprise and Solution architectures. In the interests of learning Archimate I’ve attempted to model the universally well-known¬†Domain Name System in the diagram opposite. I’m not really expert in DNS either, but because DNS is a system that most Architects know *something* about, it seems like a good candidate for sharing some modelling knowledge.

The tool used to render the model was Archi. Exported as a CSV file for ease of transfer.

Hello World in Hypermedia Style

After perhaps reading too much about Hypermedia (too much hype?) I started getting the itch to do something. This quick blog is a record of the steps I took towards a Hypermedia “hello world”.¬†The main tool I used was a plug-in¬†REST client from Mozilla¬†but anything that lets you set headers and inspect responses will do the job.

For this exercise I choose three popular Hypermedia designs: HAL, Collection+JSON and Siren. Each design has a high-level of abstraction making it suitable for re-use and an active community of developers who are busily engaged with various implementations.

HAL

In the words of Mike Kelly,¬†HAL’s creator :

HAL is a format you can use in your API that gives you a simple way of linking

See more of the HAL specification. HAL is available in either JSON or XML and has a registered media-type of

application/hal+json

Mike has documented the HAL design by providing an interactive HAL Browser. This is the most immediate way to say “hello world” in Hypermedia style. Firstly you use the HAL browser to create an account (hint: see the hints :-)). Next you should navigate to /users/:account¬†and then use the NON-GET button to get into the following dialogue box.

Saying it
Saying Hello World

It’s okay if you type something other than “hello world” *yawn* but changing anything else might break something .. To verify your creation navigate to the latest posts¬†and as-if-by-magic your entry will appear (fingers x).

The HAL browser is great way to quickly get a feel for what HAL is all about. Armed with this knowledge I wanted to take a step towards a HAL client that could be controlled independently, and ultimately deployed in another context. The HAL browser seemed tightly coupled to the server, so rather than unpick it, I dug around and found another server running on the HAL builder site. I made the following request using my trusty REST client.

GET http://gotohal.net/restbucks/api
Accept: application/hal+json

And in case you weren’t¬†paying attention earlier, here’s a picture.

Image

The HAL builder dutifully obeyed my request (yes, the Accept header is necessary) and gave me the following response.

{
    "_links": {
        "self": {
            "href": "/restbucks/api"
        },
        "orders": {
            "href": "/restbucks/api/orders"
        }
    },
    "name": "RestBucks Store",
    "project": "restbucks",
    "version": "0.0.1"
}

Now if I was a robot, I would crack-on and apply some automation to the above. Here are our first clues about Hypermedia. Custom media-types and machine-based connections! There’s way more discussion about using HAL¬†in the ¬†forum.

Collection+JSON

Mike Amundsen is the author of Collection+JSON and says it is

a JSON-based read/write hypermedia-type designed to support management and querying of simple collections

And just like HAL, Collection+JSON has its very own registered media-type. To get a test response from Collection+JSON we’ll return to our REST client. As before set the client up for the target service and specify the media-type.

GET https://graphviz-ewf.rhcloud.com
Accept: application/vnd.collection+json

This Hypermedia server responds with the following.

{
  "collection": {
    "version": "1.0",
    "href": "http://graphviz-ewf.rhcloud.com:80/",
    "links": [
      {
        "href": "http://graphviz-ewf.rhcloud.com:80/",
        "rel": "home",
        "prompt": "Home API"
      },
      {
        "href": "http://graphviz-ewf.rhcloud.com:80/graph",
        "rel": "graphs",
        "prompt": "Home Graph"
      },
      {
        "href": "http://graphviz-ewf.rhcloud.com:80/register",
        "rel": "register",
        "prompt": "User Register"
      },
      {
        "href": "http://graphviz-ewf.rhcloud.com:80/login",
        "rel": "login",
        "prompt": "User Login"
      }
    ]
  }
}

Graphviz is open source graph visualization software. The Graphviz API was designed to allow new users to register, add graph definitions and retrieve those definitions in various representations (pdf, jpg, png, gif). From the¬†example API requests¬†that are available in the Graphviz documentation, this looks like a well-conceived implementation of Collection+JSON. As authentication is required to use the Graphviz service I decided to continue my tour and found a Collection+JSON Browser. The browser functions much like the HAL browser except that the Hypermedia server is de-coupled from the client. The example¬†Employee data is open¬†and this means we can hit the endpoint directly from .. yep that’s right our friend the REST client.

Employee data in C+J representation
The above example shows a response from the Employee test data. Like HAL, Collection+JSON has a strong developer community.

Siren

Let’s complete our excursion with the newest design reviewed here. Kevin Swiber describes Siren¬†as:

a hypermedia specification for representing entities

And like HAL and Collection+JSON, Siren has a registered media-type.

application/vnd.siren+json

To make our “hello-world” request to Siren we’ll use a Siren Browser¬†developed by Wurl. Incidentally it’s great to see commercial API providers, such as Wurl and Graphviz embracing Hypermedia designs (this is the future :-)). Let’s point this Hypermedia client at a test Hypermedia server¬†that Kevin has running on heroku. As this service is read only (404, “Cannot POST /users”) we cannot use it to make a “hello world” but the action for creating a user seems clear enough from the response to the initial GET.

GET http://siren-alps.herokuapp.com/
accept:application/vnd.siren+json

Here is the response. It has been trimmed down to the essential bit.

{
    "actions": [
        {
            "name": "user-add",
            "href": "http://siren-alps.herokuapp.com/users",
            "method": "POST",
            "fields": [
                {
                    "name": "user",
                    "type": "text"
                },
                {
                    "name": "email",
                    "type": "text"
                },
                {
                    "name": "password",
                    "type": "password"
                }
            ]
        }
    ]
}

Like the others, the Siren gang also hang out and share stuff.

Final thoughts

Something that all of the three Hypermedia designs have achieved is that at no point was it necessary to Read-The-Fscking-Manual. This is a Very Good Thing for Developer Experience. This is another clue for our understanding of Hypermedia design. Self-discovery! As said earlier, each design also has a high-level of abstraction. The flexibility provided by this abstraction does however make design selection seem rather arbitrary. Which of the three Hypermedia designs are right for me?!? I hope this will be the subject of my next excursion. Thanks.

API Design for large hierarchies

Introduction to the Problem

Some things that you may, or may not know about a Wolverine.

{
    "aliases": [
        "glutton",
        "skunk bear"
    ],
    "range": "620 km",
    "weight": "9-25 kg",
    "topSpeed": "48km/h",
    "lifestyle": "Solitary",
    "lifeSpan": "10-15 years",
    "conservationStatus": "Threatened",
    "habitat": "Mountainous regions and dense forest",
    "culture": "Michigan is known as the Wolverine state",
    "claimToFame": "In order to steal a kill a wolverine attacked a polar bear and clung to its throat until the bear suffocated."
}

Given that the the taxonomy of the living world contains around 8.7 million species. The challenges presented by this wolverine snippet are all about catalog and index management. The navigation of a large hierarchy is therefore something that API designers need to address when designing their interfaces. REST defines constraints that help us organise resources, but REST isn’t aligned to any particular data model. We therefore need to work out how best to manage hierarchical data ourselves. This post is based on experience from the design of a working system.

Limitations of the client

Rather than tackle the tree of life, I will use a tree of “things” to help model the API design. But firstly we need to understand two important limitations of our imaginary clients.

  1. The maximum payload our client can manage is 512 kilobytes, i.e. half-a-meg.
  2. The client is pretty dumb and will not be able to traverse the hierarchy without assistance.

Introducing a tree of things

The total number of things in our fictitious tree is 258. Things are nested down to a depth of three. Each thing contains exactly six other things. The following diagram shows a slice of the tree.

a diagram showing payload hierarchy

The things are named using the following convention.

A1		B10 .. B16		C100 .. C106
A2		B20 .. B26		C201 .. C206
... 
A6 		B60 .. B61 		C600 .. C601

In the tree of things the smallest payload that a node can return is 96 kilobytes. Doing the maths and the total size of the model works out at 4.03 megabytes (4,128 kb).

level kilobytes numOfItems
A 96 6
B 576 36
C 3,456 216
4,128 258

In the common scenario the client would access the API with the following request.

GET /things/A/B/C/204

And the server would respond as follows:

{
    "id": "C204",
    "items": [ 1, 2, 3, 4, 5, 6 ]
}

We are pretending that each item is 16 kilobytes and so we’re happy that the total 96 kilobytes is well within the required payload limit. This okay for the happy path but as-it-stands the design exposes a lot of loose endpoints. For example, what should the server do with the following request?

GET /things/

The server cannot respond with everything in A, plus all the B things together with their C descendants because the payload would be 4.03 mb, i.e. the full hierarchy. Perhaps a more reasonable response would be to remove the B and C descendants leaving just those in the range A1 to A6. Hmm, but now we’re starting to make assumptions about the request .. let’s play safe for now and just tell the client they asked for too much.

GET /things/ 				# 416 Requested Range Not Satisfiable

Using this approach I can complete my API design.

GET /things/A/B/C/204 		        # 200 Success 
GET /things/A 				# 416 Requested Range Not Satisfiable
GET /things/ 				# 416 Requested Range Not Satisfiable
GET /things/A/1/B/10		        # 200 Success
GET /things/A/3 			# 200 Success

At this stage the design is functional as it satisfies our minimum payload criteria. But it isn’t that easy to navigate. The top-level responses are blocked, and those responses that are returned are simply a flat list. It’s hard for my dumb client to know what to do with these. A response like this would be an improvement.

{
    "id": "A3",
    "items": [
        {
            "id": "B30",
            "items": [
                {
                    "id": "C300",
                    "items": [

	                    /* more items here */
                    ]
                },

                /* and more here too */
            ]
        }
    ]
}

Although this creates a payload problem (it weighs in at 688 kb) it shows promise because I can start to educate my client about the nature of the hierarchy.

Using depth to control the payload

To help the client get to know the tree of things without breaking the payload, I add the following parameters to my design.

GET /things/A/3/A/B

This meaning of the additional A/B parameter is to instruct the server to give me the descendants of B, as well as the list of A items that were discussed previously. Here’s the response.

{
    "id": "A3",
    "items": [
        {
            "id": ["B31", "B32", "B33", "B34", "B35", "B36"]
        }
    ]
}

Effectively I’ve filtered out C and thus got my payload down to 112 kilobytes. The client has a response that matched the request and thus enough information to start the descent into the hierarchy.

Using hypermedia to improve on the 416 method and help discovery

The new controls allow the client to control the depth of the nested response but there is still room for improvement. If the client initially goes for data that is out of the bounds of my payload limit, then the server must still return an error.

GET /things/A/3/A/B/C 				# 416 Requested Range Not Satisfiable

After receiving the 416 response they have to try again by trimming the depth back to A/B. But how can my dumb client figure this out from a 416 status code? This is where  HATEOS can help! As the server knows the payload limits it can construct compliant URLs and pass those onto the client. For example.

{
    "id": "A3",
    "links": {
        "next": "/things/A/3/A/B",
        "prev": null
    }
}

Using the links part of the response we can now return a 200 whenever the request has gone out-of-bounds. The links redirect the client towards the part of the hierarchy that can be reached from the current location. To achieve this the client has some simple logic to perform.

if (res.links) {
	callService(res.links);
} else if (res.items) {
	renderItems(res.items);
} else {
	// panic!
}

In summary, the features of the API design.

  • Allow the depth of the response to be specified in the request.
  • Return links rather than errors when the requested payload size is excessive.
  • The navigational links should be sensitive to the current location in the hierarchy.
  • The links communicate to the client the maximum depth that a resource can support when providing a response.

To finish off let’s briefly return to our wolverine. Assuming that we are able to discover the wolverine endpoint through navigation, we would end up with something like the following.

GET /species/vertebrates/carnivores/weasels/wolverine

{
    "id": "wolverine",
    "items": { /* snipped wolverine facts */ },
    "links": {
        "next": [
            "/species/vertebrates/carnivores/weasels/wolverine/luscos",
            "/species/vertebrates/carnivores/weasels/wolverine/gulo"
        ],
        "prev": "/species/vertebrates/carnivores/weasels"
    }
}

The wolverine item fits ours size requirement and so we get the payload. As it turns out there are two sub-species (American and European) we get some further navigation too. It would be fun to prove this out with a really big data set and see how well the model holds up. I hope this walk-through has illustrated some of the problems and solutions surrounding API design and large hierarchies.

Common URL Pattern

When architecting with domains, services and URLs this pattern¬†seems to come round a lot, but I’ve never heard of it being given a formal name or description:

protocol. service. env. example. com / product

I will call it the Common URL Pattern. In a real-world scenario it would resolve to something like: http://www.test.example.com/support. Here is the same pattern again but this time expressed by throwing some UML around.

UML diagram showing each part embedded inside it's parent
Common Url Pattern

And finally by way of explanation, some comments on the parts.

protocol Underlying transport layer such as https, smtp
service Shared across a technology stack: www, mail, api, db
environment Instances of the stack with functional variation, e.g. test, dev
domain Identifier in the Internet namespace: a.b.c
product A website or application that handles end-user interaction, e.g. /blah

It seems like a fairly adaptable model that can be applied to a large number of websites and web-based applications.

Are WADLs yesterday’s news?

It has been four years since Sun’s Marc Hadley put forward his proposal for a Web Application Definition Language. Since the arrival of the WADL, we have seen the API Economy boom and an almost overwhelming swell towards REST and away from SOAP. Given that the WADL was intended to be the RESTful answer to SOAP’s WSDL you might have expected that the volume of WADLs would be rising in equal proportion. Somehow it hasn’t happened like that, so what’s going wrong?

The basic idea was that the WADL defined your REST interface in a way that machine generated code could consume it. After all, APIs are aimed at machines more so than humans. But a review of the clients that have actually committed code to the WADL specification suggests that outside of the Java community take-up hasn’t really happened, and even within Java it is fairly sporadic. To understand what’s going on here, let’s dig a little deeper into what a WADL currently offers.

UML diagram showing a WADL

From this UML summary of a WADL you should be able to see that a number of resources have been defined and attributed a URI, together with a method and HTTP verb. Within the resource definition are the details of the request/response cycle. For a full description try the Apigee WADL reference or the original submission to W3C .

Detractors of the WADL¬†talk about true Hypermedia clients exploring the interface through the API responses. I think this is valid. But defining the contract between the client and the server is not the only purpose of a WADL. It can also be a general description of how an API is supposed to behave, even an API that hasn’t been built yet. Commissioning an API requires the establishment of a different kind of contract; a contract between a Business Owner and an API Designer. Using a WADL to express an API design maybe a side-effect of the original purpose, however this discussion is really more about using a WADL in the context of API design than it is about implementing a REST client.

Much of the original motivation for the WADL seems to have been around providing a simple and immediate alternative to the WSDL. The fact that it hasn’t been widely adopted suggests that the specification has gone too far the other way. Is it overly simplistic? If so, what is missing – what should a future WADL specification contain in order to overcome these limitations. Here is my WADL wishlist.

  • UML and Documentation
  • Inheritance
  • Interpolation
  • Managing Application State
  • Cross-reference
  • Non-functionals
  • JSON + node.js module

UML and Documentation
Most software design is done using UML. When it comes to applying the REST architectectural style there is surprisingly little around in the UML world. For example, the popular UML tool Enterprise Architect supports only WSDLs. This is awkward when attempting to visualise RESTful designs in an Enterprise context. An experimental attempt at bridging this gap is the eaWadl tool that imports a WADL into Enterprise Architect (disclaimer I wrote it).

In addition to using the WADL as part of the design process, another good use-case is documentation. For example, the Web Application Definition Language is used by Enunciate as a framework for human-readable content. The flexible <doc> tag really helps as it allows sample JSON or XML payloads to be embedded alongside the request and/or response. A common and effective ploy is to render the WADL documentation as HTML in response to an Accept Header of text/html (which a browser sends by default).

State
With a Home Document¬†you don’t need a WADL to define the interface up-front, just grab the index and off-you-go. This is the logical extension of the Hypermedia approach. To explore this idea, let’s imagine that we have a REST interface to describe a game of chess. It should be straight-forward enough to imagine how each chess piece could be given a resource endpoint that would describe the various behaviours of each piece. For example a GET /pieces/king would provide a response that says move one square in any direction. Okay that’s fine. But what if we want to record a game of chess? This is a whole new tin of worms; that of managing the state of the game board. For example PUT /board/12/60/Qf2+. In it’s response the server should be helping the client to manage the application state by offering a link to /board/12/60. This is because only the server knows (with authority) that the game on board 12 is on the sixtieth move, that the White Queen has put the Black King in check and it is now Black’s turn.

In this scenario a WADL can be comfortably used to define the static reference data: ¬†GET /pieces/king. But because the game state is (obviously) unknown at design time, a different approach is needed. The approach should involve making general statements about the Hypermedia syntax used by the API to maintain state. The request and response elements offered by the current WADL don’t constrain how the syntax should be defined. But I would go further and argue that it should. Elements such as <id>, <rel>, <class> and <href> have become so baked into Hypermedia designs that they deserve an exclusive place in our new generation WADL.

Cross-reference
A general issue with APIs is that of API discovery. How are you supposed to find out about the existence of an API in the first place? And if you are managing a group of related APIs, how does the discovery of one API help you find out about the other members of your API family. Google have tackled this problem through their discovery API. But what about everyone else? As the WADL already allows us to reference external Xml Schemas, why shouldn’t we be able to reference other WADLs?

<grammars>
<include wadl="http://example.com/xyz/doc.wadl"/>
</grammars>

With over 9,000 APIs currently registered on Programmable Web cross-referencing should help bring some organisation to the API space.

Inheritance
It’s generally polite to reply to a question in whatever language was used to ask the question. Consequently it’s annoying that each <request> and <response> declaration needs a separate <representation> tag. Although the WADL is hierarchical there is no sense of a child node that inherits from the parent. Wouldn’t it be easier to declare the base representation in the root <resources> tag, and use the child nodes to override any exceptions?

Non-functionals
Functionality is one thing, but if the security features of the API disallow access then it’s game over. If our new WADL has helped you discover an API then it seems logical that some declaration of the required authentication is made at the point at discovery. For example <authn> and <authz> tags might have values such as “oauth” or “x509” together with a <profile="developer.example.com/register"> statement for humans. Similarly a <status> parameter could broadcast downtime whenever an origin service becomes unavailable.

Error Handling
While you can get a long way with HTTP status codes many error responses are specific to a particular domain. A common strategy is therefore to return a <code>200</code> to show that the underlying HTTP communication was successful and then maintain a set of API specific error codes. The status attribute of the <response> element should be able to document the existence of the bespoke messages.

Interpolation
APIs are deployed in multiple environments. In reality the <resources> tag is likely to resolve to a number of hosts.

  • http://dev.example.com/api
  • https://example.com/api

Rather than relying on an external tool (such as Maven) it should be possible to abstract the hostname for use in multiple environments.

JSON + node.js module
With NodeJs currently the most second popular download on github, the march of JSON seems set to continue. WADLs are of course expressed using XML. Frameworks such as deployd¬†illustrate the immediacy of using Node to define and create a REST service. With a little work, it’s possible to start this process from a WADL. With a bit more effort it would be possible to encapsulate the WADL import process using an Node module, similar to the work underway at RestDoc.

In summary, we should stick with the WADL not particularly because of it’s original intentation as a machine-readable definition of a service, but because the process of designing and developing APIs is lacking a reference language. The WADL just seems like a sensible place from which to embark.

API Predictions


This is a fantasy view of the future inspired by talking with people at APIdays. It’s a personal view and best read as science-ficiton, but you never know, some of it may come true!

NOW

Predictions in this section seem fairly likely to happen if not happening already

  1. A few branded, skyscraper APIs will continue to be dominant and used by the vast majority of apps. The number of skyscapers will represent a tiny minority of the total APIs publically available.
  2. The dx (developer experience) movement will get into full swing and impact the API design space.
  3. Hypermedia will formalise itself as a series of Architectural Design Patterns. See [1], [2] and [3] for examples.
  4. HTML5 and websockets will spawn a new generation of real-time APIs

NEXT

These predictions are not happening yet, but reasonably likely to come into place

  1. Apigee will have been acquired.
  2. Twitter will win back their developer community.
  3. NodeJs applications such as deployd [4] will continue to drive down availability of XML representations. XML may disappear from the majority of public APIs, but will continue to live-on in bespoke partner and private APIs.
  4. ProgrammableWeb will become obsolete. Instead there will be a new generation of API discovery tools [5]
  5. There will be a significant increase in the number of stand-alone API providers such as Twilio. (Twilio is a stand-alone API provider because it doesn’t have a core product in the way that Google, Facebook and Twitter do.)
  6. An open-source and independent solution for API traffic management will become available, perhaps based on [6]. API clients will become more adept at self-managing load.
  7. APIs will provide text/html as their default content-type (because humans need to understand them before the machines can get started).
  8. There will be a single go-to place for API developers to social network (and it won’t be Facebook).
  9. API security will change radically in response to the general availability of personally available hardware/mobile tokenisers.
  10. Commercial monitoring tools will get-in on the API act. Their solutions will provide views of how a single request passes through the technology stack. The apiKey will provide the glue.

FUTURE

Predictions that will probably never happen, perhaps this is a wish-list ..

  1. W3C will publish an API standard that is largely driven by Hypermedia and the requirements of the API clients. Kin Lane [7] will be involved!
  2. Semantic Web will align with a new generation of media-types that arise from the rising popularity of the Hypermedia style. A standard representation for RDF and JSON will emerge, driven by a skyscaper API provider and (eventually) blessed by W3C.
  3. A famous legal battle will project the issue of API provenance into the media. Digital signature solutions will evolve and adapt themselves to the API economy.
  4. Single Page Applications (SPAs) will be the accepted alternative to traditional page-based web sites. SPAs will use registered media-types.
  5. The Great-Twitter-Betrayal will give rise to a credit-rating style system. This system will make promises to Venture Capitalists and make them feel better about manging risk in the API economy. Everyone else will ignore it.
  6. Software tools for API management (see [8] and [9]) are provided out-of-the-box by the majority of cloud vendors.
  7. Delays in the transition from HTTP 1.1 to HTTP 2.0 will engender the adoption of a new protocol that will be optimised for API traffic and messaging.
  8. A single supplier will capture the market for API developer portals

References


[1] http://stateless.co/hal_specification.html

[2] https://github.com/kevinswiber/siren

[3] http://amundsen.com/media-types/collection/

[4] http://github.com/deployd

[5] http://www.apihub.com/

[6] http://loader.io/

[7] http://kinlane.github.io/talks

[8] http://webshell.io/

[9] https://www.runscope.com