Skip to content

Category Archives: Technology

Things Technological

Bootstrapping AWS with Terraform and CodeCommit

A rough model that I’ve been working on and thinking about recently is for the AWS account (or accounts) be put together so that there’s a “bastion” or “bootstrap” instance that can be used to build out the rest of the environment. There is a certain chicken-and-egg around this, particularly if you want to use […]

Creating a custom Kylo Sandbox

I had a need – or desire – to build a VM with a certain version of NiFi on it, and a handful of other Hadoop-type services, to act as a local sandbox. As I’ve mentioned before, I do find it slightly more convenient to use a single VM for a collection of services, rather […]

Smoke testing Kafka in HDP

Assuming that you have a vanilla HDP, or the HDP sandbox, or have installed a cluster with Ambari and added Kafka, then the following may help you to smoke test the behaviour of Kafka. Obviously if you’ve configured Kafka or Zookeeper to be running on different ports, this isn’t going to help you much, and […]

Lies, Damned Lies and Programmers

I recently came across a really nice set – not directly related – of articles dealing with various profound errors that programmers and system designers fall into when dealing with names and addresses. The TL;DR if you don’t read these: names and addresses are hard and most things you believe about them are wrong. Let’s […]

A Demonstration NiFi Cluster

In order to explore NiFi clustering, and NiFi site-to-site protocol, I decided that I could use a minimal installation – as I’m really just exploring the behaviour of NiFi itself, I don’t need to have any Hadoop environment running as well. To this end, my thought was that I could get the flexibility to just […]

List-o-mania

I have, once again, felt stuck, spinning my wheels in the mud. There is an unpleasant, and possibly vicious, cycle at play here in my head: my planning falls apart, I feel like I am not getting anything done, my anxiety spikes, I cannot plan cogently. Repeat and repeat and repeat like some damned overwrought […]

Two-factor in the middle of the night

Wherever possible I have been enabling two-factor authentication and similar protections. Not that I am paranoid, it’s just that I am paranoid. One of these I have had in play for a long time is protection on my Google account. So it’s somewhat comforting to get an unexpected SMS message from Google in the middle […]

Henry VI, Part 2, Act 4, Scene 2

The first thing we do, let’s kill all the lawyers. I’ve heard back from my contact at Blue Point regarding the defunct electric car chargers. On 4th of August I was assured by Berkeley’s “Head of Estates” that matters were going to be soon resolved, and that a quick trip through Legal would see it […]

On Git Submodules…

Git Submodules. Just Say No. Not Even Once. http://slopjong.de/2013/06/04/git-why-submodules-are-evil/ https://codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/ http://somethingsinistral.net/blog/git-submodules-are-probably-not-the-answer/

Docker and Consul and DNS, oh my

I’m still trying to wrap my head around networking when it comes to Docker and related technologies – I think because a lot of the documentation and examples around are either not quite correct, or are subtly out of date. I’ve noticed too that a lot of the writing out there around setting up Docker and/or […]