Skip to content

Elephants and Pigs

Since I did have HomeBrew installed, I went ahead with this set of instructions, with some variation. Note that at the time I did this HomeBrew installed Hadoop 2.6.0, not 2.4.x as described at this site:

https://www.getblueshift.com/setting-up-hadoop-2-4-and-pig-0-12-on-osx-locally/

I had previously verified that I could ssh to my localhost, so skipped over that, and went straight to this:

brew install hadoop
cd /usr/local/Cellar/hadoop/2.6.0/libexec/etc/hadoop
cp mapred-site.xml.template mapred-site.xml
vi hdfs-site.xml core-site.xml mapred-site.xml yarn-site.xml

And got the files looking like this:

<!-- hdfs-site.xml -->
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>
<!-- core-site.xml -->
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>
<!-- mapred-site.xml -->
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>
<!-- yarn-site.xml -->
<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
</configuration>

Following along with the instructions at the original site, I did the following, all of which seemed to work as expected (note that there are some errors in the original site, which I’ve corrected below)

cd /usr/local/Cellar/hadoop/2.6.0
./bin/hdfs namenode -format
./sbin/start-dfs.sh
./bin/hdfs dfs -mkdir /user
./bin/hdfs dfs -mkdir /user/robert
./sbin/start-yarn.sh
./bin/hdfs dfs -put libexec/etc/hadoop input
./bin/hadoop jar libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'
./bin/hdfs dfs -get output output
cat output/*
./bin/hdfs dfs -rm -r /user/robert/input/*
./bin/hdfs dfs -rm -r /user/robert/output/*
./sbin/stop-yarn.sh
./sbin/stop-dfs.sh

This all seemed to work, however there are frequent warnings which seem to be a known issue:

2015-03-09 10:13:47.495 java[11710:371646] Unable to load realm info from SCDynamicStore
15/03/09 10:13:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

As an aside, it thrills me that there are different log formats from different applications here. Since I’ve been accumulating a few bits and pieces like this, I created a pair of .command bash scripts in my local Applications folder:

#!/bin/bash
. ~/.bashrc
cd $HADOOP_HOME
./sbin/start-dfs.sh
./sbin/start-yarn.sh
#!/bin/bash
. ~/.bashrc
cd $HADOOP_HOME
./sbin/stop-dfs.sh
./sbin/stop-yarn.sh

Most of the sites talking about installing Hadoop 2.4.1 on the Mac indicate that this does not work

brew install pig

however I did that with 2.6.0 and it seems to have successfully installed Pig 0.14.0. I then adapted the experiments at (using 100Mb file initially rather than 1Gb)

http://ericlondon.com/2014/08/01/hadoop-pig-ruby-map-reduce-on-osx-via-homebrew.html

to verify that pig was, well, piggy. Specifically, what I did was copy his Ruby script and play with it. Note that article does not indicate where the data is coming from, so I used the following after creating the script:

mkdir Sandbox/pig
cd Sandbox/pib
vi map_reduce.rb
chmod +x map_reduce.rb

# Test the script outside hadoop
cat /usr/share/dict/words | ./map_reduce.rb --map | sort | ./map_reduce.rb --reduce

# create a json file with the script, then test with PIG
~/Applications/hadoopStart.command
./map_reduce.rb --create_json_file
$HADOOP_HOME/bin/hdfs dfs -mkdir input
$HADOOP_HOME/bin/hdfs dfs -rm -r output
$HADOOP_HOME/bin/hdfs dfs -put data.json input
pig
grunt> json_data = LOAD 'input/data.json' USING JsonLoader('first_name:chararray, last_name:chararray, address:chararray, city:chararray, state:chararray');
grunt> store json_data into 'output/data.csv' using PigStorage('\t','-schema');
grunt> quit
$HADOOP_HOME/bin/hdfs dfs -cat  output/data.csv/.pig_header >data.csv
$HADOOP_HOME/bin/hdfs dfs -cat output/data.csv/part* >> data.csv

Doing that did reveal some problems. In that nothing was working. A detour through the bowels of the logs showed that there were all sorts of network problems, with some parts of the system trying to get to ‘localhost’ and other parts to the apparent hostname. After flailing around, I went back to the stage of

./bin/hdfs namenode -format
./sbin/start-dfs.sh
./bin/hdfs dfs -mkdir /user
./bin/hdfs dfs -mkdir /user/robert

and then re-running the pig test, which seemed to sort it out. I think it’s all predicated on the assumption that the name of the host does not change, and I’ve noticed that Mac OS X has a tendency to change the name somewhat arbitrarily. I can probably get around this by assigning a fixed name as an alias to localhost, and altering the Hadoop configuration to use that fixed name. And sacrificing a chicken at the altar to the elder gods of the network.

ADDENDUM spoke to soon, the pig test threw a weird message, but does still seem to create the output:

2015-03-09 12:31:25,901 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2015-03-09 12:31:26,904 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-03-09 12:31:27,909 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-03-09 12:31:28,914 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-03-09 12:31:29,919 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-03-09 12:31:30,925 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-03-09 12:31:31,927 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-03-09 12:31:32,932 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-03-09 12:31:33,935 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-03-09 12:31:34,940 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-03-09 12:31:35,946 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Addendum to the Addendum
This problem turned out to be that the job history server was not running, so the two startup scripts were amended to start and stop that server:

#!/bin/bash
. ~/.bashrc
cd $HADOOP_HOME
./sbin/start-dfs.sh
./sbin/start-yarn.sh
./sbin/mr-jobhistory-daemon.sh start historyserver
#!/bin/bash
. ~/.bashrc
cd $HADOOP_HOME
./sbin/mr-jobhistory-daemon.sh stop historyserver
./sbin/stop-dfs.sh
./sbin/stop-yarn.sh

Ruby Tuesday

Except it’s Monday. So today I am working from home in order to bootstrap my brain quickly up into a better understanding of Pig. First order of business being to install it locally. A quick Google and I find a number of resources talking about how to install Hadoop and Pig, two of the top three involving using HomeBrew:

 

  1. https://www.getblueshift.com/setting-up-hadoop-2-4-and-pig-0-12-on-osx-locally/
  2. http://amodernstory.com/2014/09/23/installing-hadoop-on-mac-osx-yosemite/
  3. http://blog.shelan.org/2014/10/how-to-install-hadoop-standalone-2xx-on.html

Of course, the first thing I test is whether I have HomeBrew installed:

Rozencrantz-2:~ robert$ brew
/usr/local/bin/brew: /usr/local/Library/brew.rb:
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby:
    bad interpreter: No such file or directory
/usr/local/bin/brew: line 26: /usr/local/Library/brew.rb: Undefined error: 0

Sigh. This remains the big flaw with software as bazaar rather than software as cathedral: it is a frequent experience that sitting down to do anything will first involve a run-around getting tools fixed and replaced before actually starting.

Fortunately it looks like there’s a known work around so we will see how that goes:

  1. http://apple.stackexchange.com/questions/153790/how-to-fix-brew-after-its-upgrade-to-yosemite

Addendum:
That seems to have worked. Since I’d at some point in the past installed Brew the right way (for whatever values of Right Way you choose), then it as a matter of doing the following:

cd /usr/local/Library
git pull origin master
brew update

After that it was possible to see that brew.rb now has the correct hash-bang:

#!/System/Library/Frameworks/Ruby.framework/Versions/Current/usr/bin/ruby -W0

Cheaper Than Therapy

I’ve finally – thanks to Delia seeing something on Facebook – gotten a lease on a space that I can use as a workshop. It’s a few minutes bike ride up the Thames, in a converted shipping container (or rather two, joined together). The rent is reasonable, and the space is probably more than I really need. It’s pretty rudimentary and will probably be freezing in winter, but it’s definitely A Shed. I’ve moved all my tools in, and started settling.

To begin with, I’ve purchased one of the Sjöbergs benches, and spent Saturday afternoon assembling it (Today I went up and oiled it, and started moving tools into the storage). I’ve had Bo up there, and we’ve put his travel crate in one corner so that he can make himself at home. He seemed pretty happy there, so I can see that this will be somewhere we will stroll up to in the evening for a few hours pottering, being blokes together in the shed.

IMG_2120_2

Workbench

I did have a large workspace under my house in Australia, but paradoxically did not get to use it terribly much for myself. I fell into the pattern of doing a lot of work for other people, almost always pro-bono, and was knocking out stuff in bulk out of rubbish timber. I also fell into the pattern of buying more and more equipment so that I could do a lot of repetitive work quickly, all of which I disposed of at cut rates before coming to the UK.

  • But making saw dust is one of the reasons I wanted to come here, so that I had the chance to work with better timbers, to try to reproduce pieces I see in museums, to make things for my own pleasure, and enjoy making things again. To that end, I’m not going to be going mad purchasing a lot more equipment, but will be doing (almost) everything by hand. I did retain all my router bits, and will pick up a router soon – the ability of a router to cut rebates and grooves, and shape edges, is something I would not forgo for larger pieces of work. I also held onto the Leigh jig, for much the same reason.  I’ve also got the inevitable cheap electric drill (which we grabbed here) because making small holes with a brace is just a plain nuisance.

I will acquire some additional hand tools, though, to round out what I have. I’ve already grabbed a couple of Veritas carcass saws, which will see me right for most small stuff I’ve got in mind, but I am going to need to acquire some decent planes (which may be an expensive proposition, unless I can find some antiques in good condition).

Short term plans are pretty basic. First projects, necessarily are

  • a couple of work benches, for assembly, and to mount the Leigh jig on
  • Some shelves
  • Boxes for the shelves to put tools etc in.
  • Clean and rehabilitate tools

The boxes / shelves / tool wrangling will all have to happen more or less at the same time, but the benches are a priority, so I will have to find some way of getting some plywood and framing timber delivered (I’ve got a few different plans for some quick benches requiring not much in the way of lumber). I do have a concrete list of stuff I need to get into the shed, which I will put here so I can remember it:

  • Vacuum Cleaner (We have one that is now redundant, because it was rubbish)
  • Notebook and pens and pencils
  • Garbage can
  • Power adapter for Australian to UK mains
  • Water bowl for the dog
  • fine sandpaper for cleaning tools
  • Rags. Lots of rags for wiping stuff and hands.
  • Parafin wax
  • Brush for sweeping off sawdust.

The perfect (sword) bag

One day I will be happy, because I will have found the perfect bag to carry everything in. Meanwhile, I remain miserable, making do with a mixture of bags.

The problem really is not that there is not a range of good bags for carrying HEMA gear. The problem is one of context. Sometimes I need to carry gear for two people, and the trip varies between “putting stuff in the car” through to “getting on a plane”, with “lugging stuff through the London Tube” somewhere in the more frequent middle.

I know that every time I’ve mentioned sword – or HEMA – bags on line, the usual response is a chorus of people telling me that I’m doing it wrong. I’m sure that will happen again, and my only available response is to ask the reader to try to walk in my shoes.

I would pick four bags as being parts of perfection. Between these it looks like I can cover off most of my requirements with minimal friction.

To start with are the simple “military” echelon bags. You know the ones – made of heavy canvas, usually black or khaki, cost #10. These comfortably carry an AP jacket, fencing breeches, a mask, a gorget, and some gloves – for one person. They’re a convenient size and shape for all this semi-soft gear, small enough to balance on your lap on the tube, or tuck into a corner with no more than the usual number of exasperated sighs from commuters. What they are lousy for is carrying anything more than a few hundred meters – you can sort of put them on your back, which is really uncomfortable; or carry them by the straps, and limp along at an angle trying not to keep bumping your leg; or sling them over your shoulder and have every second A-type personality in the tube corridors knocking it off.

 

5E1CE72AA99F59D933F1DBFB689A8F13

The Sportube is a mighty fine piece of kit for the most travel, although it’s too big to fit in a Fiat Panda. I can get two rapiers, two wasters, two federschwert, two masks, two jackets, a bunch of gloves, gorgets and all the other bits and pieces all in the Series 3 tube. This is superb for flying, as it just goes in with the rest of the snowboards and skis, and sort of ok for commuting on the tube. The advantage for the commute is that it’s big enough to make other commuters sod off out of the way. The disadvantage for the London commute is that the Tube is all about stairs (god help anyone trying to get around London by public transport), and getting the fully laden SporTube up or down stairs is really a two-person job. It’s not bad for walking a fair distance with, but that does depend on the surface you are walking on – cobble stones are not fun.

The alternative to putting everything in the SporTube is a large NorthFace duffel bag. It’s like an echelon bag crossed with an SUV: it is huge, and is fitted out with straps and handles that make it comfortable to carry quite long distances on your back in back-pack mode. The handles also make it easy to swing up onto your back without knocking over fellow commuters. It’s probably too big to hold on your lap or under your feet in the train, but not so big that you can’t stand in a corner. The optimal load-out for this is two jackets, two gorgets, two masks, a bunch of gloves, two breeches, and maybe some other small bits and pieces. That keeps the weight manageable. The only drawback to these types of bags is that there’s no partial unpack: inevitably you have to get everything out in order to get to the bottom layer once you have two people’s gear in it. Another hassle with these bags is that it’s fiddly to get the top open all the way without loosening the backpack straps all the way off.

 

ASTE61P

The final bag is a new one, the Lichtenauer. This is an odd hybrid, and definitely designed by HEMA folk. It’s a very flexible design, and I can get a bunch of swords, gloves, gorget and mask for one person in it. I can definitely get two rapiers, two federschwert and two wasters in it. It’s designed to be sort of ok on your back, but feels precarious, as though it is going to slip off. That’s illusory though, it is designed when hanging on your back to sit at a bit of an angle so that you can walk. It’s not amazingly comfortable with a mask in it, but it’s ok and definitely not uncomfortable or painful. I would not be keen to use it for flying, but it’s good for the commute. It would also fit well in the Fiat, but realistically I can just shove everything in the car wrapped in a blanket just as efficiently.

So there you have it. Various possible modes. What seems to be working best for two people, on the tube, is to use the SwordBag and the NorthFace bag – swords and gloves in the SwordBag, masks and jackets and stuff in the NorthFace bag. For flying, the SportTube and either two echelon bags, or the NorthFace, depending on how many other clothes I need to take.

I will repeat my caveat, this is a statement about what works for me, for particular contexts.

First World Problems

So we have installed Rocki units in three rooms in the flat – the lounge, the bedroom, and the library. I just started playing a Clannad album from my laptop to the speakers in the lounge. Much to my bewilderment, a moment later a different album started playing in the bedroom.

I thought it may have been coming from my phone, so shut that down. Kept going. From my iPad? Shut that down. From my partner’s phone? Shut that down. Something bizarrely broken with AirFoil on my laptop? Shut down the laptop. Has someone managed to hack our network and is pranking us for lulz? Is it the NSA? MI5?

Nope. One of the cats had sat on the stereo remote control in the bedroom and started playing a CD.

I need a simpler life.

A Java Development Manifesto

I wrote this some years ago, mainly aimed at our java devs, but I think it comes close to my personal manifesto for coding in general.

1. Small Pieces, Loosely Connected.

In order to attain the sort of performance we need from (our product), and in order to be in a position to respond quickly and with agility to changes in the business, we cannot build and deploy monoliths. Our architecture and performance design is predicated on being able to scale out the service by horizontally scaling pretty well all components. Each component should be self contained, must not be dependent on the internal state or implementation of another component, and be able to be deployed into the smallest and simplest run-time environment possible. Assume we will be deploying dozens of instances of any component. Assume that we may have different versions of the same component running. Assume that we will be doing continuous deployment as well as continuous integration.

Separation of concerns is a concept sometimes applied to the construction of code, but seldom effectively applied to the construction of entire systems. We should build components that provide very specific and easily defined services, and that aim should be carried down through the layers all the way to objects. If at any point we describe a module or component using ”and”, we have a candidate for decomposition.

2. Keep it Simple. Small is Beautiful.

Small components are more easily maintained, tested, and modified. Large things are more fragile. Aim for things that can be described on a napkin, and designed on a whiteboard.

3. Test Everything.

Test your code. Test your designs. Test your implementation plans.

Test your assumptions and beliefs and habits.

Build test harnesses, unit tests, integration tests, performance tests. Play in sandboxes.

4. No Broken Windows.

All code contains bugs, and coding is the art of writing bugs. That is no excuse for not fixing bugs. Addressing the small, unimportant, niggling inconveniences wherever possible is as valuable as pushing out new features: every broken window fixed improves the quality of the product. Spending the time to fix a time-consuming annoyance saves time in the long run.

Don’t check in code that breaks the build. Let me repeat that. Don’t check in code that breaks the build.

5. Thread Safety Matters.

Assume that any data, including private member data, can be simultaneously accessed by or modified by multiple threads. Always consider the implications of that. Don’t be afraid to use the thread safety tools provided in the language, and don’t assume that state violations only occur under high concurrency conditions.

6. Optimise When You Need To.

The system as a whole has aggressive performance targets. That does not mean that all components have the same performance requirements. Don’t waste time in premature optimisation: remove performance bottlenecks that are found in testing. Profile your code.

On the other hand: assume crossing boundaries is expensive. The cheapest and quickest access is in local memory. Reading from disk, from a database, from a remote web service can be assumed to be an order of magnitude or more slower than memory access. The implication is: cross boundaries as little as possible.

7. Finish the Job.

You are not here to write code and tests. You are here to design, to communicate, to teach, to explore, to deploy, implement, refactor and attempt to destroy. Ask yourself, every time, what the relevant definition of “done” is. And when you are done, wipe down the benches, sweep the floor and put away the tools.

8. Boldly Refactor.

Go read the canonical books, then come back with a chainsaw. If you are scared to refactor because you don’t know what will break, your tests are inadequate. If it does break, you can always revert the code to the last checked in version.

9. Today is a Good Day to Die.

Work as though you may fall under a bus at lunchtime. Your code will have deficiencies that need to be corrected, and will be modified when requirements change. This means that somebody else, possibly a future you, will read and modify code. Make it easy for them, and strive to ensure they won’t curse your name.

10. Plan for Failure

You know what your code is supposed to do: build tests that try to break it. Then go back and stop it breaking. Assume the callers of your code will pass rubbish in, and horribly misuse it. Program accordingly. Assume if you call out to some other system or module that the call will fail, and that you will get rubbish back. Program accordingly. The disk will fill, the database will explode, and the server will crash. Deal with it.

This is doubly true of anything going out across the Internet, or a WAN: remote service calls can be relied on to fail. Deal with it.

Woolwich to Old Street

I’ve been meaning to write up some of the routes I take through London. Or Mordor, as a cycling acquaintance calls it, which adds some frisson of terror to the whole exercise. One does not simply ride into London. I will try to pepper this with relevant Google street view pictures, but won’t be attempting to embed maps. I find that their tools for doing that are decidedly unfriendly, and seem to assume you are working on a 24″ screen with a tablet rather than a 15″ laptop with a trackpad.

For me the route starts at Wellington Park in the Woolwich Arsenal, because our bikes are in the underground carpark beneath it. A few minutes noodling down to the Thames Path and along to the entrance of the Woolwich Tunnel. This has only recently been reopened, and touch wood it will stay that way because it’s a huge time-saver compared to the ferry. The other end of the tunnel emerges in the car park / bus stop for the ferry, and you head east initially following the A117. There is a half-hearted attempt to make a bike lane along a few meters of this, which disappears at the first corner. Be careful here, the road narrows considerably, and if you have come out at the same time that a ferry has unloaded, you get a lot of impatient lorry drivers trying to go through at the same time. I tend to plant myself in the middle of the lane for safety. Continue east along the A117 past the park on your right and over the bridges. The bike route and lane goes over the first (narrower) bridge on the carriageway, but the lane goes off just before the second bridge. Blink and you miss it, this part of the London Cycling Network (don’t laugh)

bike1

For whatever reason, this is the Sir Steve Redgrave Bridge. The end of the London City Airport is quite close to the bridge, so you can get an uncomfortably close view of the bottom of planes as they come in to land. You continue north (did you know you’d turned north?) to the giant roundabout, following the somewhat intermittent signage that shows you the London Cycle Route – the route continues on the sidewalk here, across several sets of traffic lights to get around this roundabout – don’t try to ride on the carriageway here unless you have a death wish. You want to keep north, and keep going up the A117

bike2

The bike route really is along that very overgrown path beside the road – it’s rare to meet anyone coming the other way, but exciting when you do. I have reentered the carriageway at the lights and gone around this corner, and several times have had irate BMW drivers leaning on their horns because I’m in the way – the turn is narrow, and vehicles tend to go around it fast. The bike path rejoins the carriageway as a bike lane just around this corner, and is reasonably good although a bit narrow. You continue along the A117 tending North West toward Beckton, through three roundabouts. The first two are ok, and usually very quiet, although they do the usual thing of disappearing the bike lane 20 meters before the roundabout and reappearing it 20 meters after. For, you know, magical teleporting bikes. The lane stops entirely as you go through the third roundabout, at Beckton itself. There are a few contradictory signs pointing the cycle route as going in a variety of directions – ignore them, they are all entirely wrong and will get you killed. (I suspect they intended bikes to go off to the left on a bike path just before the roundabout, across the pedestrian crossing, and then into the green space beside the A117, but this is a dead end and there is no route through).

Fortunately the road on the other side of this roundabout is quite wide, even if the cycle lane is narrow and intermittent, so you can nip along quite nicely to the enormous confusion that is the Newham Way intersection. This is where you join the Cycling Superhighway 3 (CS3) and need to start watching for blue. It’s also the point where you turn west and actually start heading toward the City. All of this will have taken you 20-30 minutes.

Now, if you glance at a map, you might ask a fairly obvious question: when you come out of the tunnel, why not go west immediately and head off up the A112, North Woolwich Road, and Silvertown Way to get to Canning town and join the CS3 there? Noodling along south of the airport, on the map, looks like a conveniently short route, doesn’t it. The trouble is, most of the A112 and area around there has been consumed by the Crossrail project, and what remains is one way, the wrong way, and very crowded and narrow. It’s just not possible to cycle it. Additionally the roundabouts near the entrance to the airport are not great: they are the hunting grounds of fare-starved taxi drivers. This is frankly bloody annoying, because there’s quite good off-road cycle routes running along the DLR route to West Silvertown, and the road route from their to Canning Town is not bad.

The other thing that might have occurred to you is to ask why when you come off the Steve Redgrave bridge you can’t go across the top of the airport, through the Excel Center. Yeah, bloody good question. You can’t. The bike paths around the University of East London don’t connect to anything at all (you cannot even ride to the huge sports centre at the roundabout there) and the bike path that should exist along the top of the water there does not. At all.  There may be some tricks for getting through this area without going along the A1020, but I’ve not yet found them. I’ve tried the A1020, and my recommendation is don’t. It is spectacularly bike unfriendly and I believe dangerous, particularly in poor light.

The CS3 from near Beckton will take you all the way in with little hassle, and is a good route, as long as you can find it. Mostly you can find it by looking for the blue path, or the blue bicycles painted on the road, but in a few patches it more or less disappears. The first section nips along to Canning Town station off the road, although there are frequent road crossings you have to make which makes it slow. The first point that it really disappears is at Canning Town station itself. You need to go left to the main pedestrian and bike crossing, then go right and over the top of the station to pick up the blue path again until it disappears completely near East India Docks. The path meets Leamouth Road, and the signage seems to indicate you should keep straight ahead – don’t, because the marked route disappears under the Crossrail works again. Instead you have to stay on the footpath and go left down to the pedestrian crossing, and through the hole in the wall

bike3

Even then things are not straight forward – the replacement path that was put in place since the Crossrail ate the main path has in turn been replaced. Up until a few weeks ago you would go straight ahead through Sorrel Lane, but that is blocked. Instead stay on the footpath and go down Leamouth Road toward the big roundabout (Leamouth Circus), and turn right up Saffron Avenue, through the boom gate and past the parked buses. The markings have now almost entirely disappeared, and there are no signs. What there are instead are some tiny discrete blue diamond painted on the paving in the nice courtyard to your left as you pass the buses that lead you across the courtyard and under the DLR through the brick arches. Turning onto this paving is an adventure if it’s been raining, there is always a very deep puddle filled with sharks and eels. As a consolation, there’s usually a coffee van parked here in the morning that does quite nice coffee. You’re going to need it for the next bit.

After a few minutes sitting in very nice surroundings with a cup of coffee, you go through the arches and under the DLR. This is still the CS3, and you will find the CS3 markings, but first you have to ride over a large cobbled area. It’s not clear who thought it was a great idea to put cobbles on a cycling super highway, and I would rather like to meet them and discuss the matter with them. Riding over this is something like having a jack-hammer up your butt.

Keep following the CS3 signage, which leads you up to and along Poplar High Street. This looks a bit mad, but the traffic is always at a crawl (and theoretically limited to 20 MPH) so it’s quite safe as long as you ride assertively. You are probably in company by now, it’s around East India that the lycra heroes are appearing, given you’re only a couple of miles from the Tower now. The only things you have to worry about as you toddle along the highway now are Boris Bikes, and as you go past Tower Hamlet College, hordes of sleep-deprived students aimlessly strolling across the street.

There is one oddity as you go along Narrow Street and turn right up Horseferry Road:

bike4

Hopefully the photo above shows it, but what you have to do is cross the traffic (and cut in front of any bikes coming down Horseferry) to ride along the right side of the road while oncoming cyclists come down the left, before crossing the narrow bridge over Limehouse Link and into St James Gardens. Do stop to appreciate this garden, it’s a lovely little space that seems to get very little use, but is well kept and quite peaceful. After the park, you are taken up onto Cable Street, and the path is off the road for the rest of the way.

You have definitely entered the realm of Boris Bikes, Bankers, and worst, Bankers on Boris Bikes. There are usually a lot of riders on this path commuting in from this point, and they all seem very proud of the mile or so they will ride. You are allowed to feel grizzled and smug, having ridden something like 10 miles at this stage.

If you follow the CS3 to the end, it ends rather abruptly in the chaotic hell around Tower Gateway. There are a cluster of big roads here that are difficult to navigate on a bike, and it’s very hard to get across the lanes to turn right to go up into the City. They’re not too bad if you are trying to ride down to Westminster, but good luck and god speed if you want to get to anywhere in the City from here.  Whoever designed this route needs their backside seriously kicked, it’s inexcusable to just dump cyclists off the end of the Cycling Superhighway into this mess – once you get onto the A100 heading West here, you are stuffed, and it’s very hard to get off. To make it even worse, there are a lot of lorries and HGV along here, going quite fast, and it’s not a good space.

Instead, go right at the Crown And Dolphin, and up Cannon Street Road:

bike5

It’s not obvious before you turn, but there is a narrow bike lane along both sides here, usually with cars parked over it. Be warned, the road surface along here is appallingly bad, and is littered with pot holes and various drains that will make your day very interesting. Generally the traffic is ok, not going very fast even if fairly busy, and you will probably be travelling in a pack of other riders. Continue on, across Commercial Road, and up New Road. Watch out for pedestrians on these crossings, they tend to wander across against the lights. On your right is the Royal London Hospital, even if it looks like a bunch of rundown warehouses and apartment buildings. Go up to and across Whitechapel – this looks like a horrible intersection, but it’s actually quite safe as the traffic lights work very well and you should not have to deal with anything turning in front of you.

Having said that, I did have the adventure a few weeks ago in a pack of cyclists when another cyclist opted to drop back, pass me on my left and then turn  right in front of me. Suffice to say I used a variety of bad words, very loudly, to indicate that this was a suboptimal manoeuvre on his part. Note that you don’t go straight across here, you have to dog-leg where that white car is heading.

bike6

You’re more or less skirting Banker land now, and about to nip through a no-mans’s land between the Land Of Bankers and the Land Of Hipsters. Go left off New Road into Hanbury Street. This can be a bit hard to spot, but the bike path is marked on your left reasonably clearly – you would spot it more easily if you weren’t dodging cars here. Nip along Hanbury Street up to Commercial Street and cross it past the Spitalfields Market into Lamb Street. Just one word about the Hanbury Street leg though – it’s intermittently sign-posted as a bi-directional cycle route, but parts of it are one-way against you for cars. Coming off Lamb Street, the cycle path goes across the paved area in front of the markets, between two coyly marked dashed lines indicating the path and accompanied by a recommendation that cyclists slow down. Which is not a bad idea, pedestrians wander aimlessly throughout this.

Cross Bishopsgate (Liverpool Street station is just down to your left) and into Primrose Street up the glass canyon.

bike7

Primrose Street is particularly marked by pedestrians stepping onto the pavement while looking left or at their phones. If you have a bell, use it aggressively. If you have a horn, try not to gore people, no matter how tempting it is. You’re almost there now. Turn right up Appold Street, left into Worship and then right into Paul Street. This is another place where there is a two-way bike path but you are going against the one-way car traffic – don’t worry about it, these streets are always close to deserted. Go past The Fox, unless you are feeling thirsty (they have quite a good selection of ales, and are a very pleasant drinking spot) and curve left into Leonard Street.  This takes you down to City Road. The Silicon Round-About is one block to your right. In other words, you have arrived.

The Old Street/City Road round-about is possibly less terrifying than it looks – there are quite a lot of cyclists up and down Old Street, despite it being not very bike friendly and always quite busy, however the system of lights around here seem to work ok for cyclists. My suggestion is to do what a lot of cyclists do around here and just occupy the centre of the lane for safety. Heading down Old Street will eventually get you to around Holborn, or alternately heading down City Road will take you back toward Bank station. I am very ambivalent about this route though – the road is chaos with various competing building and road works, is always very busy, and the road surface is dreadful. On the other hand there are usually a lot of cyclists, and if you are moving in the pack you are (probably) safe. Getting down from Bank to London Bridge is… an adventure… and one that I’ll talk about some other time. My advice for getting around the City on bike is to keep off the main arterial roads as much as possible, take it slow, and have a thick skin.

Two Wheels through The City

I’ve been quite ill, again, recently, and am still not fully recovered. Certainly not recovered enough for the walk from The Tower up to City Road to be a quick one, and not recovered enough to risk that ride. So for the past couple of days I’ve been pushing my scooter through The City. Haven’t died yet, but…

There are three scourges of the trip, in ascending order of most likely to cause me injury:

 

  1. Tourists
  2. commuters reading Facebook or Twitter or whatever on their phones
  3. Bankers

Tourists aren’t too bad, because mostly they are restricted to a few places, and are visible from a way off. Their sin is a tendency to stop in the middle of a crowded pavement without any warning, or worse, to suddenly jink one direction or another. When I used to travel through outback Australia, I would see the same behaviour with flocks of emus. As you tore down the highway, you would see a flock ahead of you by the road. When they saw the car, they would try to run away from it… down the road. When you got close enough to be able to overtake them, the birds would realise that running in a straight-line was not going to get them away from pursuit, and they would turn 90 degrees. There was an even chance that they would turn away from the car, or throw themselves under it.

If smart phones, or equivalent, are around long enough in their current form, urban humans will evolve so that like plaice one eye will migrate to the top of their head. That will allow them to keep their head down over the phone, while one eye is still looking where they are going. I would like to think that the anyone falling under a bus while head down over the device is evolution in action. Dear commuter: your body awareness is lousy to start with, don’t impair yourself further.

Finally, Bankers. I do fear that one of these blue-shirted Masters of the Universe will shoulder me into a plate glass window because their double soy decaffe latte was three degrees too cold. These are the geniuses who will, on a 3 meter wide pavement, look you in the eye as you approach and play chicken rather than share the space. And if you think I’m exaggerating, I had one this morning on an otherwise empty pavement, look me in the eye and say “I don’t have to give way to you”. No, you don’t, but the pavement is 3 meters wide, I am using one meter of it, you can easily use the other 2. I know that you’re told that you only win if you take it all, but that just makes you an arsehole.

Passwords definitely considered broken

So we have news of yet another major slurping-up of poorly secured credential sets. A column at the Guardian talks about all the usual measures that can be taken to more-or-less protect your multiple identities, but once again misses the two subtle and deeply geeky issues that underly this breach.

First off, we can guess that the Russians swept up a combination of user name / password pairs from various sources where the user name was in plain text, and the password was either in plaintext, or hashed. At best they were salted hashes, but in all probability they were simple MD5 hashes. The real risk that comes out of this is that this sort of volume of data feeds into the construction of rainbow tables and the more sophisticated techniques that entirely circumvent mindless brute force attacks. The more often this happens, the more likely it is that hashed passwords will be able to be converted back to plain text in the future. Having super long complicated passwords, as the article at the Guardian suggests, is not going to protect you if the site you have entered them on is storing them as plain text or unsalted hashes, or even as simplistically salted hashes. You will get some protection by using different passwords in different places, but if someone chooses to attack you and subverts key passwords like a Google or Facebook account, there is every likelihood they could amplify their attack using those core accounts.

(By the way, if all the talk of hashes and salted hashes and plain text and cypher text is bewildering you, I can strongly recommend Simon Singh’s “The Code Book” as an entertaining and very readable history of cryptography and survey of most of the current crypto methods now available. Also, wikipedia)

The real core problem remains that user name / password pairs are a terrible security mechanism. Even before time-stressed and semi-competent coders do something with user name / password pairs, it’s a lousy mechanism. And the things that these code monkeys will do with the credentials are very predictable: chuck them in a database table named something like ‘account’ or ‘password’ or ‘user’. If you are lucky they will MD5 encrypt the password rather than leaving it in plain text, but they probably won’t. A slightly more experienced coder will do a salted encryption, but because they’ve used your email address as a user name, it’s not real hard to reverse that. A smarter coder will use the underlying Unix password mechanisms, and cross their fingers that the system administrator has got the box protected sufficiently (this is actually reasonably secure, and we’re now getting into the realm of sophisticated hacks – as long as the system administrator has not left all the doors wide open). If they’re really thinking, they will use something like LDAP, but that’s hard, and relies on good infrastructure, and good system administrators, and starts costing money, and oh my god we’ve got to get this site live by Thursday, we will fix it later…

Yeah, good luck with that. You have to hope that the company who hired some cheap code monkey to bang out the website you are now setting up an account on  didn’t go for the lowest bidder.

The real solution is that we technologists have to tear down this whole lazy, half-arsed default assumption that we will have a user-name/password pair.  For a start, could we please start separating the difference between ‘identity’ and ‘authorisation’. And for God’s sake, when you are in a meeting with the client and discussion of the user-name/password entry required to order a pizza on line, leap across the table and throttle whoever is insisting that they really really really need that. Virtually none of the places I have had to create accounts actually need accounts in order to allow me to do business with them. It’s just lazy habit.

For authorisation, we’ve got any number of alternatives, such as OAUTH and OpenID, or even Facebook’s horrible and intrusive federated authorisation system. If you really, really want to have some sort of login, please outsource it to someone who knows what they are doing. Longer term, let’s get away from user names and passwords all together. What you are really trying to do is ask two questions: who is on the other side of the keyboard, and are they allowed to do what they are trying to do. Let’s go with biometrics. Let’s go with two-factor authentication. Let’s go with anything other than freaking username/password pairs.

And in all seriousness, if you are in the position of having any input to the design of (particularly) web-based systems, push back strongly on the requirement for the user to have identified themselves to transact or read the site. A good model is Amazon’s actually – you can faff around endlessly on their site, and you only need to eventually identify yourself when it comes time to provide payment details. An even better option that I’ve seen in a very few places is to ask at that point whether the user wants to ‘checkout as guest’ and allow them to provide whatever account details they want. Seriously, if the user is happy to type in their name and shipping address every time rather than having to commit to creating a user-name/password protected account, why stop them?

An Open Letter to Australia.

If you believe adult asylum seekers are not, entitled to their claims, to they should be “sent back to where they came from”, or they are “queue jumpers”: If you do not say to the elected Australian Government that this is wrong, then you are tacitly supporting this treatment of children

Would you like your children, or your nieces and nephews, or your friends’ children to go through this?

All that is necessary for the triumph of evil is that good men do nothing .