Skip to content

Zookeeper on Raspberry Pi

Ok, yeah, I probably don’t really need a Zookeeper cluster running in the house, but there are a few Raspberry Pi’s around that are mostly idle — we use them for streaming music from our various devices to the speakers in some of the rooms.


Thibault – with added diagrams

We heard that Thibault liked diagrams, so here are some diagrams about Thibault’s diagrams…

Some years ago my partner-in-crime (and life) and I presented an introductory workshop on Thibault, and as part of preparing for that, they drew up some flowcharts for the relationships between the “circles” in Thibault’s plates, from Chapter 5 through Chapter 8.


Wasabi and AWS S3 – A comparison

Wasabi is a very interesting and compelling competitor for AWS S3… but also potentially a superb collaborator.

What are Wasabi and S3 though? Stripping these services down to their barest bones, they are cloud-based, highly available and resilient object stores with effectively unlimited storage capacity. Digging a bit further, they are both key/value stores, which means that every binary object is uniquely identified by a key, in much the same way that a file on your laptop is uniquely identified by a folder path and file name.


Lies, Damned Lies and Graphs

It appears from discussion at Wikipedia that the catchphrase “lies, damned lies and statistics” is in fact unattributed. That’s a shame, because it’s a pretty important idea – statistics are very slippery, and in this time of COVID-19 I’m seeing how easily they can be misunderstood, and misused.


AWS EC2 Instance Connect — A very neat trick

One of the problems with cloud security compared to on-premise is that there is more risk that someone unauthorised will be able to gain access to your EC2 linux instances via SSH. That’s one of the reasons I’m keen on server less solutions, various X-As-A-Service services, and on not opening up a server for access by SSH at all. It’s easier to keep bad guys off a server if you don’t let anyone onto the server.



A reasonably common scenario for a data-focussed consultancy is that a client may want to ship sensitive data from their on-premise or cloud environment to your AWS environment. There are a number of reasons that they may want to copy the data into your environment: it may be difficult for you to work with it in-situ, the tools you need may not be inside their environment, their may be no ingress to their data stores from outside, or they may want to provide an extract of data rather than the raw sources. These are all valid scenarios under which the simplest scenario is to be able to dump the sensitive data into an S3 bucket under your control.


More Swarm Adventures

I recently went back to refresh my understanding of the state of Docker networking (there’s been some changes over the last few years I wanted to be sure of), and so have been working through the excellent tutorial materials they have built, and spinning off some tutorial materials of my own demonstrating automation of the setups.

For your interest, here’s a Terraform project on AWS that sets up a Docker Swarm to play with – of course in reality we’d use ECS and EKS, but this is a fun exercise in infrastructure-as-code:

Adventures with Docker Swarm

It’s been around 3 years since I last worked with Docker in any seriousness. At that time, the state of networking and deployment was quite rudimentary, and there was still reliance on deploying load balancers and similar infrastructure. I was very impressed then, when revisiting the “getting started” tutorials, at how straight-forward and powerful Docker Swarm now is.

I’ve built a small implementation of those tutorials to illustrate the ease with which a full stack can be deployed.

Four Questions For Engineers

One way of looking at the art and science of software engineering is that it is a process of mapping human desires and wishes — the insides of peoples’ heads — onto a computer system. This is not a particularly novel idea, and it’s one that you are probably familiar with, but it’s an important one. Engagement with a client can be boiled down to a conversation wherein we discover the client’s needs and wishes, and then present an instantiation of our interpretation of what they have expressed. There is an awful lot of chance for error in this. Mapping the contents of their heads to vibrations in the air and symbols on paper or a screen is a lossy process. Our interpretation of what we hear or read is a lossy process. Implementing the ideas, dreams and wishes into an information system is a lossy process. It’s a wonder software ever gets built at all.


Cross-Account use of AWS CLI

The documentation around using the AWS CLI from an AWS EC2 instance on one account to access resources in another account are not great. The information is all there, somewhere, but it’s scattered across many places and to derive what you need from those sources you have to pretty well read all the sources. Two useful places to begin, but you will need to spiral out from, are:

However, I’ll try to give a summary and simple example here. This won’t include code or detailed instructions to set this up, although I hope to follow this up with a code demonstration expressed in Terraform.