AWS, Minecraft and Bungeecord

This is an account of how mc.fnarg.net was setup on Amazon as a distributed cluster of Minecraft nodes.

CREATING THE INSTANCES ON AMAZON
We logged into the AWS console and went to the EC2 console. Try https://console.aws.amazon.com/ec2/ and you should be redirected to a region.
Then we created two t2.micro instances. Each instance was created with an image size of 8GB. The linux distro was Ubuntu 16.04 LTS.

Both instances are in the same security group. Check that the VPC ID and Sub ID are the same for both instances. It should be similar to:

VPC ID Sub ID
vpc-34bf1523 subnet-e6543xyz

To help follow the story, the instance names and private IP adresses are shown below:

mc-gateway: x.x.x.164
mc-gamer: x.x.x.30

mc_fnarg_net
To complete the setup we connect to each instance as follows.

ssh -i ~/.ssh/aws.pem ubuntu@mc.fnarg.net

In this case we are using a Mac, you can use Putty on Windows.

INITIAL SERVER SETUP AS UBUNTU USER

Once we are logged on then we prepared each instances by installing some prerequisites.

sudo apt-get update
sudo apt-get install git
sudo apt-get install openjdk-8-jre-headless

Then we checked the java install using

ubuntu@ip-x-x-x-30:~$ java -version
openjdk version "1.8.0_131"

Next we added a minecraft group and user. This just adds a bit more security by separating process from the ubuntu user.

sudo addgroup --system minecraft
sudo useradd -g minecraft -m minecraft
sudo su - minecraft

And finally we started our favorite window manager to allows us to keep things running once we logout.

script /dev/null
screen

Screen needs the script command because we have done su.

SETUP A NODE AS MINECRAFT USER

We just follow the instructions on the Spigo site
https://www.spigotmc.org/wiki/buildtools/

mkdir /home/minecraft/build
cd /home/minecraft/build/
curl -o BuildTools.jar https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar
git config --global --unset core.autocrlf
java -jar BuildTools.jar

After a while the build process completed, so we had a look at what it did.

minecraft@ip-x.x.x-30:~/build$ ls -1F
apache-maven-3.5.0/
BuildData/
BuildTools.jar
BuildTools.log.txt
Bukkit/
CraftBukkit/
craftbukkit-1.12.1.jar
Spigot/
spigot-1.12.1.jar
work/

When Spigot upgrades we will do this again, so to keep things neat we run the minecraft server somewhere else.

mkdir /home/minecraft/server
mv spigot-1.12.1.jar ../server/spigot.jar
cd ../server/

Next we copy a start.sh from Spigo and fix the permissions. We have to edit server.properties too.

online-mode=false

This is because our clients will connect via the gateway. And on the mc-gateway node only, we change the port so we don’t clash with the gateway.

server-port=25566

Run ./start.sh once, update the eula.txt and we’re done.

SETUP A GATEWAY AS MINECRAFT USER

The last chapter is about setting up the gateway. This is a summary of instructions at
https://www.spigotmc.org/wiki/bungeecord-installation/

First make sure we are connected to the mc-gateway instance.

mkdir /home/minecraft/gateway
cd /home/minecraft/gateway
curl -o BungeeCord.jar https://ci.md-5.net/job/BungeeCord/BungeeCord.jar

As with Spigot we need a start script to run it.

#!/bin/sh
# https://www.spigotmc.org/wiki/bungeecord-installation/
java -Xms512M -Xmx512M -jar BungeeCord.jar

In the config.yml we update the servers section to reflect our architecture.

servers:
lobby:
motd: '&1Fnarg Lobby'
address: localhost:25566
restricted: false
timeout: 30000
games:
motd: '&1Fnarg Games'
address: x.x.x.30:25565
restricted: false
timeout: 30000

And lastly we run the gateway and connect from our Minecraft client. Our lobby and game nodes are shown as options when we hit /server.

2017-09-17_20.21.36

There was some additional work to add plugins on the nodes, but that’s enough for now. Thanks for reading

Manchester Attack

The extermination of a single mosquito does not eradicate malaria
Another approach is to target the breeding grounds, rivers, swamps etcetera.

At the heart of the Manchester tragedy is a young man who decided to end his own life
Irrespective of subsequent events this itself is a tragedy
The same is true for any person who resorts to suicide.

Communities should adapt and defend against forces that lead to fragmentation
Prejudice is such a force
Society members must get better at reading the signs of alienation
All must develop more effective treatments to help those who become desperate.

This act of suicide, however was also an act of mass murder.
The simultaneous combination of these acts changes everything.

Ultimately only the murderer knows what influenced his fatal decision, but we can speculate that:
They were derived from the suicide bomber behaviour pattern
Or influences that emanated from his immediate peers and family members
Or solely from his own determination, from entirely within his own mind.

Although it is difficult to separate the external from the internal influences, they must exist
Such external influences are nothing to do with religion
Religious belief gives us reason to live, not to die or take-away life
The architects of this terrible act are not Muslim
Wearing the cloak of a Muslim does not make it so.

Within the broad church of our plural society such facades create confusion
Complex societies lack the precision with which to see imposters
To rescue religion from the parasites embedded within, we must re-focus
We need to re-align religion to society and society to religion.

Evicting the fake Muslims is necessary to constrain the influence of terror
To be a Muslim (or a Jew, or a Christian) should anyway not require access to  privileged enclaves
Such divisions do not exist amongst the victims of the Manchester Attack
Nor should not exist within the institutions of religion.

Removing terrorism demands interventions that are both micro and macro
The micro interventions are to reduce suicide through greater social cohesion
And at the macro level, the challenge is to prevent religion from being a safe harbour for terrorists.

As social cohesion is an attribute of religion then we can put it more simply and say that
To eradicate terrorism we have to promote spirituality over everything

Even materialism.

Login to CAS using Python

CAS is a protocol for implementing Single-Sign-On authentication services
Python is a programming languages often used to develop web applications

To protect a Python generated web page with a CAS session a client of the CAS server has to be added to the logic of the page. Users accessing the protected page are prompted to login unless a session already exists in their browser. There are many CAS clients written in Python and this blog is a review of them. Leave a reply if any are missing and I will take a look. cas_max_logo_0

Most CAS clients are written in the context of a Python framework. Django is one of the most popular, so we’ll start there.

The original client for django supported CAS protocols 1 and 2. Ming Chen’s client inherits from the above and added support for version 3. The same author provides some Python CAS utils for testing. Another CAS client for Django is also derived from the original client but did not add support for version 3. Jerome Leleu is that author of the pac4j Java client and provides a python demo  also based on the original. Although not a client of CAS, it’s worth mentioning that Jason Bittel has a server implementation of the CAS protocol that runs on Django.

Another Python framework is Flask and Flask-CAS is a client for that framework. The client for Twisted looks like an incomplete effort at both a client and server.

Kellen’s client is for the lightweight bottle.py framework. Bottle.py can be either Python version 2 or 3. This client was written for Python 2.7.

The remaining clients in this review are designed to work in either a CGI or WSGI context.

Ian Wright has a fairly recent client that support protocol 2, 3 and Python version 3. It is designed for an Apache context. The earliest client (2011) seems to be by Jon Rifkin and runs as a Python CGI Web app (although the install failed for me). Ryan Fox has a package of the Rifkin client for pip installers. This client can also be found adapted for WSGI and embedded in a PriceHistory project. And for completeness here is link to an ancient version that looks best forgotten.

Summary

The client that worked for me was Kellen’s client which installed easily with pip. Having a working bottle.py installation was the biggest factor influencing my choice. Ian Wright’s client was next in line and for someone with a working Django environment then the Ming Chen version looks good.

On the dangers of afternoon naps

On Sunday I mowed the lawn in the new house.
After that I had to take anti-hystamine, due to the pollen.
Feeling extremely drowsy I crashed out on the sunlounger in the garden.
In my sleep I dreamt about the last time that I took anti-hystamine.
This was last summer, when we went canoeing and I fell asleep in the canoe.
But in my dream this was due to natives on the riverside firing tranquilisers through blowpipes.
And somehow I mixed this up with what my son had said about a bush in the garden.
A bush in the garden that was next to the sunlounger where I was sleeping.
A bush in the garden that had .. what was it again, a nest of insects?
I awoke with a start in a cloud of wasps.
One of them had just stung me in the arm.
But it didn’t hurt too much, because anti-hystamine is also a form of insect repellent.

I moved the sunlounger and went back to sleep 🙂

The Great Unlenders

There is a joke in this, but first I need to set the scene. In Sept 2013 I visited my local bank in Spain, Andalusia. After subjecting my status to some minor inquisitions I departed with a quotation for a mortgage. Yay! Actually, I learnt retrospectively that the quote was not legally binding but nonetheless Our Great Property Hunt was officially underway. Sometime later, my family and I found our dream home and I returned to the bank to refresh the quotation against the agreed sale price. No problemo. Armed with the confidence of an approved budget we set out to fulfill our part of the bargain and source the deposit (forty two percent of the sale price).

dream home

This meant selling our family home back in the UK. Given that we lived in a small community with some of the most fantastic people I have ever met, this was definitely the toughest part. But we did it, paid a deposit to the Spanish seller (ten percent of the sale price) and returned to the bank to seal the deal. What happened next was a mysterious series of:

  • Computer system failures
  • Banking staff absent
  • Zero response to communications
  • Ambiguous and conflicting messages from .. Madrid

Time passed. We recruited a Spanish lawyer. A great guy, very determined with oodles of experience, but even with this mighty assistance we were still unable to find a resolution. We tried again with another bank. Much the same story, but after forty days we received this email (translated from Spanish).

The appraiser (TINSA) gives me a very low value, 20 percent of the sale price.
I understand that with this value we cannot provide the mortgage.
Thank you and regards

I contacted TINSA directly to try and understand what had happened. Like the banks, there was no response. We kept trying with other banks but it was agonisingly painful as all followed a pattern that I can best describe as a strategy of procrastination.

So what’s the problem here?

Well really there are only two possibilities. Either they don’t like my house or they don’t like me. Despite an increasing sense of paranoia I can’t accept that it is anything to do with me. None of the banks have raised any concerns about my Age, Income, Nationality or any other aspect of my status. The house is the problem. It could be that it is because the house is classified as a Casa Rustica (Country House). Of the 2,823 mortgages granted in Andalusia last month only fourteen percent were Rusticas. The remainder were Urban Dwellings.

Data sourced from the National Statistics Institute (INE).

To put those mortgage approval figures from INE into some context, In April this year, La Vanguardia reported that:

The signing of new mortgages for home purchases fell by 33% last February over the same month of 2013 and has already accumulated 46 months of declines, according to data released today by the National Statistics Institute (INE ).

But nevertheless 408 mortgages were granted to lenders buying Casa Rusticas, so why do the banks decline my business? The other possibility is that the bank doesn’t like my house because it is not owned by them.

Let’s think about that.

On the 9th June 2012, the BBC covered the Spanish bailout.

Spanish banks to get up to 100bn euros in rescue loans.

At the time, Senor de Guindos (Spain’s economic minister) said

We hope that as a result of these injections [of capital] families and companies will have more solvent banks which are able to offer them credit.

So the intervention of Senor de Guindos should have helped me right? Let’s look at some numbers. These are extracted from The Bank Lending Survey, published by the Bank of Spain.

The Bank Lending Survey is an official quarterly survey that has been conducted in coordination by all the euro area national central banks and the European Central Bank (ECB) since January 2003.

From the survey, I became interested in the following question.

Over the past three months, how have your bank’s conditions and terms for approving loans to households for house purchase changed?

The following chart shows the share of banks reporting that terms and conditions have been tightened, minus the share of banks reporting that they have been eased.

lendingHouseholds

So that’s pretty clear, the banks basically shut-up-shop after they got the bailout. Senor de Guindos’s hopes were not realised and the reason for the lockout is because the banks tightened their T&Cs. But can we go further? Does any of this show that Spanish Banks are actively obstructing mortgage applications in order to promote the sale of their own toxic stock? Well not necessarily. To know that for certain we’d need to know whether the mortgages that were approved were on properties owned by the banks. And to the best of my knowledge, the banks don’t publish figures to show that. Looking more broadly for circumstantial evidence however, The Economist wrote that

With more than 600,000 new homes still unsold, a flood of new properties onto the market would hurt both banks and house prices.

And this Bloomberg article about how the EU smiled while Spanish banks cooked the books strongly suggests that there has been little or no pressure from the EU to pass funds from the European Social Fund onto lenders (in spite of Senor de Guindo’s good intentions).

But perhaps the most compelling clue about the true destiny of those European funds is revealed by the announcement this week from Spanish news agency EFE.

Banco Sabadell earned between January and September this year a net profit of 265.2 million euros, 42.5% more than in the same period of 2013

Banco Sabadell is my bank and where my search began over twelve months ago. My family is stuck. We are anxious about the security of our ten percent deposit and as temporary residents we are not properly integrating with our local community. And now there is another family stuck behind us, waiting to move into the property that we should have vacated months ago.

Anyway, here’s that joke.

Q. At the end of the Spanish crisis who will own the bigger villa, the Estate Agent or the Bank Manager?
A. Neither, they both end up in la prisión.

Heh.

APIDaysBCN 2014

Once again I was lucky enough to attend an API Days event. This year the conference was in Barcelona. The scale of the event was noticeably increased compared to last year’s conference in Madrid. This time the venue was the Mobile World Centre in Plaza Catalunya, opposite the Bank of Spain. This in itself is perhaps an indication of the rapid evolution of the API industry.Mobile World Centre

This post is a reconstruction of my scribbles over the two days.

The API Scene

API principles

Notes from the Kin Lane, The API Evangelist.

Don’t invent the wheel – use familiar patterns. For example, an API describing a profile should consider something like /me used on Facebook.

Share API designs

Find APIs (an alternative to Programmble Web)

APIs and value chains from 3scale by Manfred

The Power of /me from Bruno Pedro. An overview of the battle between the online giants for ownership of user identities, with a quick dip into OAuth2

APIs that offer interesting Services

APIs for integrating online payments

APIs for sending emails

Examples of real-time APIs and webhooks

Twitter APIs from Romain Huet. The Twitter demo covered three topics

  1. Monitoring and filtering Tweets in real time from the Streaming APIs
  2. Tweeting pictures from a Raspberry Pi and its Camera Module
  3. Controlling a Parrot AR.Drone from Tweets and acknowledging commands

Source code from the demos is available here

Each topic is covered in more detail

Streaming is based on all the tweets, an average of ~500 million tweets per day exposed through Firehose.

Vigiglobe are using Twitter streams to analyse public opinion

Another example, the Brit Awards peaked at 78k tweets per minute. The endpoints are simple.

POST https://stream.twitter.com/1.1/statuses/filter.json
POST https://stream.twitter.com/1.1/statuses/sample.json
(links needs to be within an Authenticaton context)

The sample endpoint streams one percent of the entire Firehose. You need some serious infrastructure and commercials to consume the whole thing. The Twitter Streaming API uses long polling to hold a connection open. The Twitter recommendation is that stream consumers use a single client running on their network (something like Node.js) to harvest and filter tweets. The client would then use web sockets to push notifications out to website visitors.

Another cool thing is the Amazon integration with Twitter allows users to add items to their shopping cart through #AmazonBasket.

More information about become a certified Twitter partner

Here’s that photo taken by a tweeting Raspberry PI (I’m in there somewhere).

romainWithPi

Dropbox updates from Leah Culver

Dropbox have recently added support for datastores to their APIs. This turns a dropbox folder into a NoSQL database, located in the cloud.

The user account defines the quota. And here’s an example that uses datastore to hold game state and high scores

(2048 is a game that everyone is going crazy about .. apparently).

Other Stuff

Big data for free (as in beer)

Using Dremel an open source java implementation of Google Big Query and Redis is in there too.

Tools and Tips for API Designers

The wireshark of API sniffing (from 3scale who were conference sponsors by the way)

Slides from the conference

API design at Heroku from @brandur

Insightful review of the API design process undergone by Heroku. The APIs control access to the Heroku platform and prior to the release of v3 was extensively re-designed. The API is innovative through it’s use of JSON Schemas. Heroku have generously published the API guidelines that were developed as a result of the process.

Conference slides

Example of API that uses Hypermedia well (from Ori Pekelman)

In true Hypermedia style, this is the only published endpoint – everything else is through discovery

GET http://api.navitia.io/v1/
How to do Hypermedia well
How to do Hypermedia well

 

 

 

 

 

 

 

 

 

 

 

 

 

Example from Salesforce of a Mobile API using Angular

Optimistic API Design from Pau Ramon Revilla / Api designer @ redbooth.com

Use PATCH for partial updates

problem: two PUTs in a race condition but only one can win

PUT { "name":"Robert", "hiScore":"25", "id":"123" }
PUT { "name":"Bob", "hiScore":"35", "id":"123" }

because PUT requires the entire payload. PATCH allows a partial update.

PATCH { "hiScore":"25", "id":"123" }
PATCH { "hiScore":"35", "id":"123" }

More on PATCH. Use a 202 response for non-deterministic operations.

DELETE /things/1
202 # accepted for further process
 { "rel":"/things/1/AD234" }

… some time passes

GET /things/1/AD234
200 # deletion request processed

Ronnie Mitra from Layer 7 presented a tool for prototyping API designs. Using a drag-n-drop interface the tool allows an API designer to create API endpoints as “cells”. There is some automation for the process of creating dummy request / responses and also support for Hypermedia-style linking. Ronnie promised me that it will export to WADLs and other API specification formats.

Finally here’s a picture of the main man, KinLane.

kinLane

An idealistic solution to the Spanish housing crisis

The central government of Spain should sponsor a programme to address the problem of incomplete building works. The role of the government should be to facilitate a programme that is implemented by each of the Spanish regions. The scale of the problem differs between regions, for example Andalucia has a large number of incomplete buildings littered along the Costa Del Sol. The central government should work with the central banks to provide a loan to the region that is proportionate to the reconstruction effort. Each region would then create a backlog of developments rated according to the cost of completion.

spanish building site
An abandoned building site in Spain

The region would then sponsor the formation of housing co-operatives. Each co-operative would draw on a mixture of unemployed people and those with experience in construction. A co-operative would apply to take ownership of a building site. The criteria for awarding a site to a co-operative would be based on the following:

  • The estimated capability of the co-operative to fulfill it’s promise
  • The amount of loan required to fund the re-construction
  • A loan repayment plan based on a mixture of rental earnings and sales from the salvaged properties
  • The destruction of one or more sites that have been rated as un-salvagable

During the re-construction and destruction period, co-operative members would draw on the loans to provide a social wage and funds for building materials. As habitable properties emerge from the projects, co-operative members can re-imburse themselves through either a period of rent-free occupation, or by selling on the open market and keeping a share of the profit. The remaining income would then be used to settle the initial loan to the central bank.

In this way, all the incomplete sites would either be converted to habitable dwellings or the land made suitable for re-use. A positive side-effect of the programme would be to reduce unemployment amongst young Spanish people.

It is possible that such plans exist already, but from my reading of (English-written) media it would appear that the hopes of the Spanish government rely entirely on foreign investors. The above solution maybe idealistic but it would offer the Spanish people autonomy, and control of their own destiny.

The Domain Name System modelled according to Archimate

a diagram explaining domain names
The Domain Name System modelled according to Archimate

The Archimate modelling language is a way of describing Enterprise and Solution architectures. In the interests of learning Archimate I’ve attempted to model the universally well-known Domain Name System in the diagram opposite. I’m not really expert in DNS either, but because DNS is a system that most Architects know *something* about, it seems like a good candidate for sharing some modelling knowledge.

The tool used to render the model was Archi. Exported as a CSV file for ease of transfer.