Skip to content

Workshop, Mark II

I’ve moved my workshop to a new location, which has the advantage of security, lower cost, and a far more pleasant location. Also, apparently it’s a studio now, if only I could either monetise it or adopt the life of a penniless bohemian starving artist.


Creating a custom Kylo Sandbox

I had a need – or desire – to build a VM with a certain version of NiFi on it, and a handful of other Hadoop-type services, to act as a local sandbox. As I’ve mentioned before, I do find it slightly more convenient to use a single VM for a collection of services, rather than a collection of Docker images, mainly because it allows me to open the bonnet of the box and get my hands dirty fiddling with the insides of the machine. Since I wanted to be picky about what was getting installed, I opted to start from scratch rather than re-using the HDP or Kylo sandboxes.

The only real complication was that I realised that I also wanted to drop Kylo on this sandbox, which happened after I’d already gone down the route of getting NiFi installed. This was entertaining as it revealed various ways in which the documentation and scripts around installing Kylo have some inadvertent hard-wired assumptions about where and how NiFi is installed that I needed to work around.


Smoke testing Kafka in HDP

Assuming that you have a vanilla HDP, or the HDP sandbox, or have installed a cluster with Ambari and added Kafka, then the following may help you to smoke test the behaviour of Kafka. Obviously if you’ve configured Kafka or Zookeeper to be running on different ports, this isn’t going to help you much, and it also assumes that you are testing on one of the cluster boxes, and a ton of other assumptions.

The following assumes that you have found and changed to the Kafka installation directory – for default Ambari or HDP installations, this is probably under /usr/hdp, but your mileage may vary. To begin with, you might need to pre-create a testing topic:

    --zookeeper localhost:2181 \
		--create --replication-factor 1 \
		--partitions 1 \
		--topic test

then in one terminal window, run a simple consumer:

bin/ \
    --zookeeper localhost:2181 \
		--topic test \

Note that this is reading from the beginning of the topic, if you want to just tail the recent entries, omit the --from-beginning instruction. Finally, in another terminal window, open a dummy producer:

bin/ \
    --broker-list localhost:6667 \
		--topic test

There is an annoying asymmetry here – the consumer and most other utilities look to ZooKeeper to find the brokers, but the dummy producer requires an explicit pointer to one or more of the brokers. On this consumer window, type stuff, and you should see it echoed realtime in the consumer window. When finished, ^C out of the producer and consumer, and consider your work done.

Lies, Damned Lies and Programmers

I recently came across a really nice set – not directly related – of articles dealing with various profound errors that programmers and system designers fall into when dealing with names and addresses.

The TL;DR if you don’t read these: names and addresses are hard and most things you believe about them are wrong.

Let’s start with Falsehoods Programmers Believe About Names. Without even trying the author lists 40 things we believe about names that are just plain wrong.

In a similar vein, Falsehoods programmers believe about addresses, which particularly speaks to me. One of the fundamental errors about addresses is to think they identify a location. This is incorrect. An address might identify a location, but it is fundamentally a description which instructs a postman how to deliver a letter or parcel. Substitute pizza operative, Amazon driver or writ server as desired.

Even without getting into the weirdness around the actual shape of the planet, Falsehoods programmers believe about geography touches on place names.

And as a bonus: Falsehoods programmers believe about time – computers prove to be pretty bad clocks, and working out a calendar is very complicated.

A Demonstration NiFi Cluster

In order to explore NiFi clustering, and NiFi site-to-site protocol, I decided that I could use a minimal installation – as I’m really just exploring the behaviour of NiFi itself, I don’t need to have any Hadoop environment running as well. To this end, my thought was that I could get the flexibility to just play around that I need by building a minimal Centos/7 virtual machine, running in VirtualBox. The plan was to have little more than a Java 8 SDK and NiFi installed on this, and then I would clone copies of it which would be modified to be independent nodes in a cluster. At the time of writing this is still in progress, but I thought it was worth capturing some information about how I proceeded to get my VM prepared.

There are a handful of requirements for this VM:

  1. It needs a static IP (so that I can assign different static IPs to the clones, later)
  2. It needs to be able to reach out to the broader internet, in order to pull down OS updates and similar
  3. I need to be able to ssh to it from my desktop
  4. Different instances of the VM need to be able to reach each other easily
  5. A Java 8 JVM is needed



I have, once again, felt stuck, spinning my wheels in the mud. There is an unpleasant, and possibly vicious, cycle at play here in my head: my planning falls apart, I feel like I am not getting anything done, my anxiety spikes, I cannot plan cogently. Repeat and repeat and repeat like some damned overwrought Philip Glass piece. I am trying to look at this dispassionately, because if I can understand how this happens, maybe I can head it off next time.

There are a few factors – health, political chaos, and too many months of uncertainty at work. Having a work and personal phone, and a work and personal computer, and disconnected accounts across both is really not helping either – I keep dropping things between the various calendars and todo lists, which has been exacerbated in the last few months by traveling. You would think that separating work and non-work would be easy. I can partition off my 37.5 hours and leave it at work, can’t I? Well, no. Because I’m trying to juggle calendars and waking hours and mental effort between work and non-work, and I cannot just turn off my brain at the end of the working day. Increasingly I feel like I would do very well if I cloned myself at least twice, so that different instances of myself could live full and uncomplicated lives. And I really resent the 3+ hours tied up each day in commuting, even while I know other people are doing the same or worse.


Thibault – Chapter 6

On Attacks and Counters In the Straight Line

Zachary, in the preceding chapter, saw how easy it was for Alexander to defend the simple attacks at First Instance. He asks Alexander to give him those attacks so that he can practice. Alexander, being a jerk, tries to win the drill by introducing the subjection, to which Zachary responds with a variety of “oh my god, you have your SWORD in my FACE” reactions.


Two-factor in the middle of the night

Wherever possible I have been enabling two-factor authentication and similar protections. Not that I am paranoid, it’s just that I am paranoid. One of these I have had in play for a long time is protection on my Google account. So it’s somewhat comforting to get an unexpected SMS message from Google in the middle of the night sending me an unexpected authorisation code. Because it means whoever just tried to access my account could not.

Lock your doors people. A simple username and password combination, particularly on anything critical, is effectively useless.

Thibault – Chapter 5

On Attacks at the First Instance, and Feints

This is the first chapter where Thibault leaves off his purely theoretical discussion and begins actual paired exercises. Poor Zachary comes off rather the worse for wear here, as he launches a variety of simple thrusts straight down the diameter (with one exception) from the first instance i.e. the distance he minutely detailed in the previous chapter. At least part of the point of this chapter is to setup the reasons for the actions and plays in the next chapter, as well as illustrate that the straight line is sufficient preparation and fortification against these attacks. Having said that though, to use his own words:

…fortified against all manner of feints, assured against all attacks…always making the (counter) with small movements, which have more force than showiness, and making the execution with as much force and assurance as possible, in opposition to common practice. If you say to me that it is not likely that anyone may easily reach such perfection in demonstrating all these effects, I answer that nothing commendable can ordinarily be acquired without great labour.


If it looks like a Duck

Wikipedia has a good article on the probable source of what is now known as the Duck Test

If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.

In recent weeks, in light of what is happening in the US, and the direction various European states are moving, I was going to write a rant about using “Nazi” as a short hand descriptor. But I won’t.

Until recently, I was never particularly comfortable with any modern reactionary being labelled as a “Nazi”. Even “neo-Nazi” is something I was not keen on. My reasoning behind this is a little circuitous, and I know that some folk won’t agree with me, so bear with me a little. Over my lifetime I have seen a change in the way that “Nazis” were presented in popular western culture, and how they are present in the zeitgeist. Going back to the 70’s, I recall that we started to see movies (other than movies about WW2) where villains were identified as Nazis – I am thinking of things like Marathon Man and The Boys From Brazil, where the villain was a remnant representive of an ancient and terrible evil. Fast forward to the 80’s, and we see a more light-hearted and comic-book evil in Raiders Of The Lost Ark, or the 90’s with Doom and other video games where Nazis have transformed into “abstract Bad Guys”. Most presentations of Nazis in popular culture now are as arbitrary mooks that nobody can object to seeing punched, shot, or bombed.

This is not a new idea of mine, if you dig around you will find far better analyses of this process of abstraction, and indeed trivialisation. I encourage you to do so. So until recently, I would rather that real-world current evil bastards not be labelled with something that has become a trivial insult. I would until recently have said “look, they are not actual Nazis, and you are using that label as a lazy short hand for evil bell ends who should be locked away on an island for our own safety

Except I can’t. Not any more. If it looks like a Nazi, salutes like a Nazi, marches like a Nazi, talks like a Nazi, I think we should treat it like a Nazi. Particularly since it is evident that in the US there are significant numbers of people who are literally draping themselves with the symbology and styling of the historical Nazis.

So my plea to you now is: If you want to label someone as Nazi, please double check your reasons for doing so. We owe it to the people who fought the Nazis last time around to take the label seriously. Mostly for myself I will be trying to use “racist arsehole”, “nationalistic bell-end” and “evil twat”, because I do not want to award these pricks with any of the weight that the very specific historical name carries.