Reduction with micropayments

The environmentalists have taught us the three Rs.

banana

  • Reduce
  • Reuse
  • Recycle

Reduction makes us think about unnecessary items, such as packaging. Packaging is useful to protect items we purchase and to advertise the contents of the product.

Micropayments is a trend in consumer behaviour. Advancement in payment systems makes it common-place to purchase low-value items electronically.

micropayments

Let’s say I want to purchase one litre of cooking oil. My only choice is to purchase a plastic bottle along with my oil. This is because there is no reasonable means to transport the oil from the supermarket to my home. Taking my old bottle for a refill doesn’t help, because oil is not sold on tap. This is because the cooking oil vendor has a barcode stamped on the bottle in order to collect payment. To persuade the vendor to sell me his oil on tap I need to replace the barcode with something else.

This is where micropayments can help. With an Near Fields Communication device (NFC) attached to a dispenser the point of sale and collection can be combined. Oil can now be purchased from the dispenser using any container of my choice. Given this option, I would choose to Reduce the number of plastic bottles going from my home to landfill ūüôā

 

 

Three Brokers Compared

Introduction

I am new to trading on forex. This blog is about some of my early trading experiences and my attempt to apply some logic to the process of electing a broker. The three brokers that I have considered are Fortuneway, IG and AvaTrade.

How well can you predict the market?

The market is like the weather. The further ahead you try to predict the less likely you are to be accurate. This is particularly true of the bitcoin market. The primary decisions a trader can take is when to enter the market and when to exit. Timing is key. The other, lesser factor is unit price. We can think of this as an attribute of the forecast. It represents confidence. These are the factors that the trader can control.

Here is a summary of 62 trades that I made over August and September, 2018. The percentage of successful trades was 63%. In this case, success simply means that the profit was greater than zero.

  • The biggest loss was $14,962.95
  • The biggest win was $10,578.00 with a total profit of $16,797.00
  • The average profit was $270.92
  • The proportion of buying to selling was fifty percent
  • The unit price averaged at 0.4

The results are based on demo accounts, sadly all fake money ūüôĄ

History screen using FortuneWay demo account

What factors influence success?

As an individual trader you cannot access the market directly. Only a broker can do that. There are many brokers, each wraps trading activity with policies that are designed to earn them money, and hopefully you too. How does the choice of broker impact the trading outcome?

As we are interested in the success of predicting the market across all the brokers, we first try to normalise the data. The analysis uses the points moved between open and close. This a better metric than profit because USD is skewed due to the different ways that brokers calculate profit.

Boxplot comparing trading history with three brokers
Boxplot comparing trading history with three brokers

The diagram shows how the points were distributed across data set.

  • The red line is the median (second quartile)
  • Upper and lower lines are the first and third quartiles
  • Plus signs (+) are the outliers

Trades with FortuneWay were the most successful as all the points are above zero. Conversely the trades with IG resulted mostly in loss.

Profit calculation

Brokers affect profit as follows:

  1. Using Contract Size when converting BTC to USD
  2. Applying spread when a trade is opened

The profits were calculated from the open and close prices using this forumula.

pointsMoved = openPrice - closePrice  # assume buying
     profit = pointsMoved * (contractSize * unitPrice)

# Examples from the data set

(6379.23 - 6390.05 = 10.82) * (100 * .1) = $108.20 # Fway
(6682.07 - 6450 23 =  2.07) * (  1 * .1) =  $23.20 # IG

The brokers offer these contract sizes: FortuneWay: 100, AvaTrade: 10, IG: 1

AvaTrade claim their contract size is 10 but on my data it worked out as 7.7 

This explains why the broker with the largest contract size (FortuneWay) generated the greatest profit. This is despite the fact that they won the smallest number of points (see the boxplot diagram above).

The following table shows the contract sizes, as well as all the attributes of the policy provided by each broker.

 FortuneWayIGAvaTrade
Leverage1:1001:200
Contract Size100110 (7.7)
Stops level6005003100
Profit calcFuturesCFDCFD
Spreadfloatingfloatingfloating
Digits222
Pendings GTCYesYes
Tick value1.00
Tick size0.01
Swap typeUSD
Swap long-10.00-28.8
Swap short-10.00-28.8
Margin calcFuturesCFD
Margin initial3000.00
Margin maintenance3000.00
Margin hedge1500.005.00
Margin percentage4.00
3-days swapFriday
ExecutionMarket
Min vol.01
Max vol10000.00
Vol step.01

Attributes from MT4 Quotes > BTCUSD > Details

Understanding Spread

Spread is the difference between the ask and bid price. Spread is applied when the trade is struck. For example, IG will always add 50 points, in either direction.

In MT4 See Quotes > Advanced for the current spread.

In the screenshot below were are using the AvaTrade account and when the trade was struck the the price was 6391.86
Chart showing price when struck

Chart showing price when struck

Unlike the other brokers, the spread that AvaTrade applies will vary according to the price. From the Trade screen we can see the price that was paid at that time was 6440.57. In this example, the spread was therefore 6440.57 Р6391.86 = 48.71

Screenshot of opening price
Screenshot of opening price

Effectively this means the trade has to anticipate 48.71 points before getting into profit.

Summary

The discussion has concentrated on the opportunities and risk of profit and loss, associated with the various brokers. Let’s round-off with a higher-level summary of the differences.

 AvaTradeFortuneWayIG
Profit opportunity******
Risk management******
Regulation******
Website and support******

IG scored highest on regulation because they are backed by the FSA and I am a UK citizen. Although the FortuneWay website is under-developed, the real-time support has been excellent and they were the most profitable due to their thin spread. Avatrade have the best website, but a lack of transparency on their pricing structure lowered their overall score.

Will I ever open a Real Account and which broker will I chose? You have to wait for the next blog to find out ūüėõ

You may wish to download the data used for the analysis.

AWS, Minecraft and Bungeecord

This is an account of how mc.fnarg.net was setup on Amazon as a distributed cluster of Minecraft nodes.

CREATING THE INSTANCES ON AMAZON
We logged into the AWS console and went to the EC2 console. Try https://console.aws.amazon.com/ec2/ and you should be redirected to a region.
Then we created two t2.micro instances. Each instance was created with an image size of 8GB. The linux distro was Ubuntu 16.04 LTS.

Both instances are in the same security group. Check that the VPC ID and Sub ID are the same for both instances. It should be similar to:

VPC ID Sub ID
vpc-34bf1523 subnet-e6543xyz

To help follow the story, the instance names and private IP adresses are shown below:

mc-gateway: x.x.x.164
mc-gamer: x.x.x.30

mc_fnarg_net
To complete the setup we connect to each instance as follows.

ssh -i ~/.ssh/aws.pem ubuntu@mc.fnarg.net

In this case we are using a Mac, you can use Putty on Windows.

INITIAL SERVER SETUP AS UBUNTU USER

Once we are logged on then we prepared each instances by installing some prerequisites.

sudo apt-get update
sudo apt-get install git
sudo apt-get install openjdk-8-jre-headless

Then we checked the java install using

ubuntu@ip-x-x-x-30:~$ java -version
openjdk version "1.8.0_131"

Next we added a minecraft group and user. This just adds a bit more security by separating process from the ubuntu user.

sudo addgroup --system minecraft
sudo useradd -g minecraft -m minecraft
sudo su - minecraft

And finally we started our favorite window manager to allows us to keep things running once we logout.

script /dev/null
screen

Screen needs the script command because we have done su.

SETUP A NODE AS MINECRAFT USER

We just follow the instructions on the Spigo site
https://www.spigotmc.org/wiki/buildtools/

mkdir /home/minecraft/build
cd /home/minecraft/build/
curl -o BuildTools.jar https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar
git config --global --unset core.autocrlf
java -jar BuildTools.jar

After a while the build process completed, so we had a look at what it did.

minecraft@ip-x.x.x-30:~/build$ ls -1F
apache-maven-3.5.0/
BuildData/
BuildTools.jar
BuildTools.log.txt
Bukkit/
CraftBukkit/
craftbukkit-1.12.1.jar
Spigot/
spigot-1.12.1.jar
work/

When Spigot upgrades we will do this again, so to keep things neat we run the minecraft server somewhere else.

mkdir /home/minecraft/server
mv spigot-1.12.1.jar ../server/spigot.jar
cd ../server/

Next we copy a start.sh from Spigo and fix the permissions. We have to edit server.properties too.

online-mode=false

This is because our clients will connect via the gateway. And on the mc-gateway node only, we change the port so we don’t clash with the gateway.

server-port=25566

Run ./start.sh once, update the eula.txt and we’re done.

SETUP A GATEWAY AS MINECRAFT USER

The last chapter is about setting up the gateway. This is a summary of instructions at
https://www.spigotmc.org/wiki/bungeecord-installation/

First make sure we are connected to the mc-gateway instance.

mkdir /home/minecraft/gateway
cd /home/minecraft/gateway
curl -o BungeeCord.jar https://ci.md-5.net/job/BungeeCord/BungeeCord.jar

As with Spigot we need a start script to run it.

#!/bin/sh
# https://www.spigotmc.org/wiki/bungeecord-installation/
java -Xms512M -Xmx512M -jar BungeeCord.jar

In the config.yml we update the servers section to reflect our architecture.

servers:
lobby:
motd: '&1Fnarg Lobby'
address: localhost:25566
restricted: false
timeout: 30000
games:
motd: '&1Fnarg Games'
address: x.x.x.30:25565
restricted: false
timeout: 30000

And lastly we run the gateway and connect from our Minecraft client. Our lobby and game nodes are shown as options when we hit /server.

2017-09-17_20.21.36

There was some additional work to add plugins on the nodes, but that’s enough for now. Thanks for reading

Manchester Attack

The extermination of a single mosquito does not eradicate malaria
Another approach is to target the breeding grounds, rivers, swamps etcetera.

At the heart of the Manchester tragedy is a young man who decided to end his own life
Irrespective of subsequent events this itself is a tragedy
The same is true for any person who resorts to suicide.

Communities should adapt and defend against forces that lead to fragmentation
Prejudice is such a force
Society members must get better at reading the signs of alienation
All must develop more effective treatments to help those who become desperate.

This act of suicide, however was also an act of mass murder.
The simultaneous combination of these acts changes everything.

Ultimately only the murderer knows what influenced his fatal decision, but we can speculate that:
They were derived from the suicide bomber behaviour pattern
Or influences that emanated from his immediate peers and family members
Or solely from his own determination, from entirely within his own mind.

Although it is difficult to separate the external from the internal influences, they must exist
Such external influences are nothing to do with religion
Religious belief gives us reason to live, not to die or take-away life
The architects of this terrible act are not Muslim
Wearing the cloak of a Muslim does not make it so.

Within the broad church of our plural society such facades create confusion
Complex societies lack the precision with which to see imposters
To rescue religion from the parasites embedded within, we must re-focus
We need to re-align religion to society and society to religion.

Evicting the fake Muslims is necessary to constrain the influence of terror
To be a Muslim (or a Jew, or a Christian) should anyway not require access to  privileged enclaves
Such divisions do not exist amongst the victims of the Manchester Attack
Nor should not exist within the institutions of religion.

Removing terrorism demands interventions that are both micro and macro
The micro interventions are to reduce suicide through greater social cohesion
And at the macro level, the challenge is to prevent religion from being a safe harbour for terrorists.

As social cohesion is an attribute of religion then we can put it more simply and say that
To eradicate terrorism we have to promote spirituality over everything

Even materialism.

Login to CAS using Python

CAS is a protocol for implementing Single-Sign-On authentication services
Python is a programming languages often used to develop web applications

To protect a Python generated web page with a CAS session a client of the CAS server has to be added to the logic of the page. Users accessing the protected page are prompted to login unless a session already exists in their browser. There are many CAS clients written in Python and this blog is a review of them. Leave a reply if any are missing and I will take a look. cas_max_logo_0

Most CAS clients are written in the context of a Python framework.¬†Django¬†is one of the most popular, so we’ll start there.

The¬†original client for django¬†supported CAS protocols 1 and 2.¬†Ming Chen’s client inherits from the above and added support for version 3. The same author provides some¬†Python CAS utils¬†for testing.¬†Another CAS client for Django¬†is¬†also derived from the original client but did not add support for version 3. Jerome Leleu is that author of the pac4j Java client and provides a python demo ¬†also based¬†on the original. Although not a client of CAS, it’s worth mentioning that¬†Jason Bittel has a server implementation of the CAS protocol that runs on Django.

Another Python framework is Flask and Flask-CAS is a client for that framework. The client for Twisted looks like an incomplete effort at both a client and server.

Kellen’s client¬†is for the lightweight bottle.py framework. Bottle.py can be either Python version 2 or 3. This client was written for Python 2.7.

The remaining clients in this review are designed to work in either a CGI or WSGI context.

Ian Wright has a fairly recent client that support protocol 2, 3 and Python version 3. It is designed for an Apache context. The earliest client (2011) seems to be by Jon Rifkin and runs as a Python CGI Web app (although the install failed for me). Ryan Fox has a package of the Rifkin client for pip installers. This client can also be found adapted for WSGI and embedded in a PriceHistory project. And for completeness here is link to an ancient version that looks best forgotten.

Summary

The client that worked for me was Kellen’s client which installed easily with pip. Having a working bottle.py installation was the biggest factor influencing my choice. Ian Wright’s client was next in line and for someone with a working Django environment then the Ming Chen version looks good.

On the dangers of afternoon naps

On Sunday I mowed the lawn in the new house.
After that I had to take anti-hystamine, due to the pollen.
Feeling extremely drowsy I crashed out on the sunlounger in the garden.
In my sleep I dreamt about the last time that I took anti-hystamine.
This was last summer, when we went canoeing and I fell asleep in the canoe.
But in my dream this was due to natives on the riverside firing tranquilisers through blowpipes.
And somehow I mixed this up with what my son had said about a bush in the garden.
A bush in the garden that was next to the sunlounger where I was sleeping.
A bush in the garden that had .. what was it again, a nest of insects?
I awoke with a start in a cloud of wasps.
One of them had just stung me in the arm.
But it didn’t hurt too much, because anti-hystamine is also a form of insect repellent.

I moved the sunlounger and went back to sleep ūüôā

The Great Unlenders

There is a joke in this, but first I need to set the scene. In Sept 2013 I visited my local bank in Spain, Andalusia. After subjecting my status to some minor inquisitions I departed with a quotation for a mortgage. Yay! Actually, I learnt retrospectively that the quote was not legally binding but nonetheless Our Great Property Hunt was officially underway. Sometime later, my family and I found our dream home and I returned to the bank to refresh the quotation against the agreed sale price. No problemo. Armed with the confidence of an approved budget we set out to fulfill our part of the bargain and source the deposit (forty two percent of the sale price).

dream home

This meant selling our family home back in the UK. Given that we lived in a small community with some of the most fantastic people I have ever met, this was definitely the toughest part. But we did it, paid a deposit to the Spanish seller (ten percent of the sale price) and returned to the bank to seal the deal. What happened next was a mysterious series of:

  • Computer system failures
  • Banking staff absent
  • Zero response to communications
  • Ambiguous and conflicting messages from ..¬†Madrid

Time passed. We recruited a Spanish lawyer. A great guy, very determined with oodles of experience, but even with this mighty assistance we were still unable to find a resolution. We tried again with another bank. Much the same story, but after forty days we received this email (translated from Spanish).

The appraiser (TINSA) gives me a very low value, 20 percent of the sale price.
I understand that with this value we cannot provide the mortgage.
Thank you and regards

I contacted TINSA directly to try and understand what had happened. Like the banks, there was no response. We kept trying with other banks but it was agonisingly painful as all followed a pattern that I can best describe as a strategy of procrastination.

So what’s the problem here?

Well really there are only two possibilities. Either they don’t like my house or they don’t like me. Despite an increasing sense of paranoia I can’t accept that it is anything to do with me. None of the banks have raised any concerns about my Age, Income, Nationality or any other aspect of my status. The house is the problem. It could be that it is because the house is classified as a Casa Rustica (Country House). Of the 2,823 mortgages granted in Andalusia last month only fourteen percent were Rusticas. The remainder were Urban Dwellings.

Data sourced from the National Statistics Institute (INE).

To put those mortgage approval figures from INE into some context, In April this year, La Vanguardia reported that:

The signing of new mortgages for home purchases fell by 33% last February over the same month of 2013 and has already accumulated 46 months of declines, according to data released today by the National Statistics Institute (INE ).

But nevertheless 408 mortgages were granted to lenders buying Casa Rusticas, so why do the banks decline my business? The other possibility is that the bank doesn’t like my house because it is not owned by them.

Let’s think about that.

On the 9th June 2012, the BBC covered the Spanish bailout.

Spanish banks to get up to 100bn euros in rescue loans.

At the time, Senor de Guindos (Spain’s economic minister) said

We hope that as a result of these injections [of capital] families and companies will have more solvent banks which are able to offer them credit.

So the intervention of Senor de Guindos should have helped me right? Let’s look at some numbers. These are extracted from The Bank Lending Survey, published by the Bank of Spain.

The Bank Lending Survey is an official quarterly survey that has been conducted in coordination by all the euro area national central banks and the European Central Bank (ECB) since January 2003.

From the survey, I became interested in the following question.

Over the past three months, how have your bank’s conditions and terms for approving loans to households for house purchase changed?

The following chart shows the share of banks reporting that terms and conditions have been tightened, minus the share of banks reporting that they have been eased.

lendingHouseholds

So that’s pretty clear, the banks basically shut-up-shop after they got the bailout. Senor de Guindos’s hopes were not realised and the reason for the lockout is because¬†the banks tightened their T&Cs. But can we go further? Does any of this show that Spanish Banks are actively obstructing mortgage applications in order to promote the sale of their own toxic stock? Well not necessarily. To know that for certain we’d need to know whether the mortgages that were approved were on properties owned by the banks. And to the best of my knowledge, the banks don’t publish figures to show that. Looking more broadly for circumstantial evidence however, The Economist wrote that

With more than 600,000 new homes still unsold, a flood of new properties onto the market would hurt both banks and house prices.

And this Bloomberg article about how the EU smiled while Spanish banks cooked the books¬†strongly suggests that there has been little or no pressure from the EU to pass funds from the European Social Fund onto lenders (in spite of Senor de Guindo’s good intentions).

But perhaps the most compelling clue about the true destiny of those European funds is revealed by the announcement this week from Spanish news agency EFE.

Banco Sabadell earned between January and September this year a net profit of 265.2 million euros, 42.5% more than in the same period of 2013

Banco Sabadell is my bank and where my search began over twelve months ago. My family is stuck. We are anxious about the security of our ten percent deposit and as temporary residents we are not properly integrating with our local community. And now there is another family stuck behind us, waiting to move into the property that we should have vacated months ago.

Anyway, here’s that joke.

Q. At the end of the Spanish crisis who will own the bigger villa, the Estate Agent or the Bank Manager?
A. Neither, they both end up in la prisión.

Heh.

APIDaysBCN 2014

Once again I was lucky enough to attend an API Days event. This year the¬†conference was in Barcelona. The scale of the event¬†was noticeably increased compared to last year’s conference in Madrid. This time the venue was the¬†Mobile World Centre in Plaza Catalunya, opposite the Bank of Spain. This in itself is perhaps an indication of the rapid evolution of the API industry.Mobile World Centre

This post is a reconstruction of my scribbles over the two days.

The API Scene

API principles

Notes from the Kin Lane, The API Evangelist.

Don’t invent the wheel – use familiar patterns.¬†For example, an API describing a profile should consider something like¬†/me used on Facebook.

Share API designs

Find APIs (an alternative to Programmble Web)

APIs and value chains from 3scale by Manfred

The Power of /me from Bruno Pedro. An overview of the battle between the online giants for ownership of user identities, with a quick dip into OAuth2

APIs that offer interesting Services

APIs for integrating online payments

APIs for sending emails

Examples of real-time APIs and webhooks

Twitter APIs from Romain Huet. The Twitter demo covered three topics

  1. Monitoring and filtering Tweets in real time from the Streaming APIs
  2. Tweeting pictures from a Raspberry Pi and its Camera Module
  3. Controlling a Parrot AR.Drone from Tweets and acknowledging commands

Source code from the demos is available here

Each topic is covered in more detail

Streaming is based on all the tweets, an average of ~500 million tweets per day exposed through Firehose.

Vigiglobe are using Twitter streams to analyse public opinion

Another example, the Brit Awards peaked at 78k tweets per minute. The endpoints are simple.

POST https://stream.twitter.com/1.1/statuses/filter.json
POST https://stream.twitter.com/1.1/statuses/sample.json
(links needs to be within an Authenticaton context)

The sample endpoint streams one percent of the entire Firehose. You need some serious infrastructure and commercials to consume the whole thing. The Twitter Streaming API uses long polling to hold a connection open. The Twitter recommendation is that stream consumers use a single client running on their network (something like Node.js) to harvest and filter tweets. The client would then use web sockets to push notifications out to website visitors.

Another cool thing is the Amazon integration with Twitter allows users to add items to their shopping cart through #AmazonBasket.

More information about become a certified Twitter partner

Here’s that photo taken by a tweeting Raspberry PI (I’m in there somewhere).

romainWithPi

Dropbox updates from Leah Culver

Dropbox have recently added support for datastores to their APIs. This turns a dropbox folder into a NoSQL database, located in the cloud.

The user account defines the quota. And here’s an example that uses datastore to hold game state and high scores

(2048 is a game that everyone is going crazy about .. apparently).

Other Stuff

Big data for free (as in beer)

Using Dremel an open source java implementation of Google Big Query and Redis is in there too.

Tools and Tips for API Designers

The wireshark of API sniffing (from 3scale who were conference sponsors by the way)

Slides from the conference

API design at Heroku from @brandur

Insightful¬†review of the API design process undergone by Heroku. The APIs control access to the Heroku platform and prior to the release of v3 was extensively re-designed. The API is innovative through it’s use of JSON Schemas. Heroku have generously published the API guidelines that were developed as a result of the process.

Conference slides

Example of API that uses Hypermedia well (from Ori Pekelman)

In true Hypermedia style, this is the only published endpoint – everything else is through discovery

GET http://api.navitia.io/v1/
How to do Hypermedia well
How to do Hypermedia well

 

 

 

 

 

 

 

 

 

 

 

 

 

Example from Salesforce of a Mobile API using Angular

Optimistic API Design from Pau Ramon Revilla / Api designer @ redbooth.com

Use PATCH for partial updates

problem: two PUTs in a race condition but only one can win

PUT { "name":"Robert", "hiScore":"25", "id":"123" }
PUT { "name":"Bob", "hiScore":"35", "id":"123" }

because PUT requires the entire payload. PATCH allows a partial update.

PATCH { "hiScore":"25", "id":"123" }
PATCH { "hiScore":"35", "id":"123" }

More on PATCH. Use a 202 response for non-deterministic operations.

DELETE /things/1
202 # accepted for further process
 { "rel":"/things/1/AD234" }

… some time passes

GET /things/1/AD234
200 # deletion request processed

Ronnie Mitra¬†from Layer 7 presented a tool for prototyping API designs. Using a drag-n-drop interface the tool allows an API designer to create API endpoints as “cells”. There is some automation for the process of creating dummy request / responses and also support for Hypermedia-style linking. Ronnie promised me that it will export to WADLs and other API specification formats.

Finally here’s a picture of the main man, KinLane.

kinLane