The Occasional Masthead

In order to explore NiFi clustering, and NiFi site-to-site protocol, I decided that I could use a minimal installation – as I’m really just exploring the behaviour of NiFi itself, I don’t need to have any Hadoop environment running as well. To this end, my thought was that I could get the flexibility to just play around that I need by building a minimal Centos/7 virtual machine, running in VirtualBox. The plan was to have little more than a Java 8 SDK and NiFi installed on this, and then I would clone copies of it which would be modified to be independent nodes in a cluster. At the time of writing this is still in progress, but I thought it was worth capturing some information about how I proceeded to get my VM prepared.

There are a handful of requirements for this VM:

1. It needs a static IP (so that I can assign different static IPs to the clones, later)
2. It needs to be able to reach out to the broader internet, in order to pull down OS updates and similar
3. I need to be able to ssh to it from my desktop
4. Different instances of the VM need to be able to reach each other easily
5. A Java 8 JVM is needed

Installing Centos

This then is how I proceeded.

To start with I downloaded a mimimal ISO from https://www.centos.org/download and installed it as a VM based on the suggestions at http://www.jeramysingleton.com/install-centos-7-minimal-in-virtualbox. The install process is pretty straight forward, and about the only interaction needed is to set the root password. Since I prefer to not login as root, I also created a centos user at this stage and marked it as an administrator. At this stage, I had only a single VirtualBox network adapter assigned to the VM, configured as a NAT network (this is the default), so I went ahead and set up port forwarding for ssh access in VirtualBox:

• Protocol = TCP
• Host IP = 127.0.0.1
• Host Port = 2222
• Guest IP = 10.0.3.15 (This can be found by executing ip addr in the VM)
• Guest Port = 22

This then allowed me to ssh to the box from my local machine:

$ ssh -p 2222 centos@localhost

Because I’d marked the centos user as an administrator during installation, it has sudo access. By default this access requests a password each time, which I find annoying because I am lazy. To prevent this, you need to edit /etc/sudoers to ensure that it has the setting

%wheel ALL=(ALL) NOPASSWD: ALL

which states that members of the group wheel do not have to provide a password for any sudo invocations.

Next step was to bring the box up to date, and install some quality of life tools. Additionally I noticed that postfix was running by default in this box, so I removed it:

[root@localhost ~]# yum update
[root@localhost ~]# yum install psmisc net-tools bind-utils
[root@localhost ~]# systemctl stop postfix.service
[root@localhost ~]# systemctl disable postfix.service

then halted the VM and made a snapshot – I did that at several points through the setup, in order to have good stable places that I could revert to if I mucked anything up.

Next step is to install Java 8, which I did by following https://tecadmin.net/install-java-8-on-centos-rhel-and-fedora. Again, this is quite straight forward – download the package from Oracle (going through their very annoying web form), and then fiddle about with alternatives to get the magic links in /usr/bin setup. I did run into a problem though:

[root@localhost ~]# java -version
bash: /bin/java: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

Google-fu led me to this: https://janardhanareddy.wordpress.com/2014/09/16/centos-7-java-installation-problem-bad-elf-no-such-directory and i blindly did yum install zlib.i686 which resolved the issue.

I also added an /etc/profile.d/java.sh file to get the path into all users’ environments:

# java environment setup
export JAVA_HOME=/opt/jdk1.8.0_131
export PATH=$JAVA_HOME/bin:$PATH

All of this took about an hour, most of it waiting for stuff to download over the office network. Next step should have been easy, and turned out to take the rest of the afternoon and a good deal of coffee: adding a static IP address.

Setting up the networks

My first thought seemed like a reasonable one – I’ve already got a virtual network adapter set up as NAT, I can add an additional virtual network adapter as Host-only Adapter using the default vboxnet0 network, then modify the network configuration in the VM to make use of this adapter. I did this, then all sorts of pain arose: depending on what I did, access to the box was denied, or the box would not reach out to the broader internet, or ssh to the box would hang for up to a minute. I rubber ducked with Calvin Hartwell for a while around this, successfully managing to make two people confused and bewildered. Finally I turned it off and then on again (more or less). It turns out that what I needed to do was:

1. alter the initial default NAT adapter to be Host-only
2. edit the /etc/sysconfig/network-scripts script to specify the fixed IP information
3. add an additional network adapter attached as NAT.

Doing it this way meant that my IP network (in the VM) looks like:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen
1000
    link/ether 08:00:27:25:9a:0e brd ff:ff:ff:ff:ff:ff
    inet 192.168.63.101/24 brd 192.168.63.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet6 fe80::492b:45cf:4a5d:cfc6/64 scope link
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen
1000
    link/ether 08:00:27:46:31:3f brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.15/24 brd 10.0.3.255 scope global dynamic enp0s8
       valid_lft 84314sec preferred_lft 84314sec
    inet6 fe80::f351:f4d4:d42a:1458/64 scope link
       valid_lft forever preferred_lft forever

In other words, the box has a fixed IP address in the 192.168.63.??? network, and a NATted address sorted out by the DHCP service in VirtualBox. Weirdly, if the two adapters are reversed, i.e. enp0s8 was the 192.168.63.??? address, everything went haywire. I can only conjecture that this was related to either some unidentified configuration that I was missing, or that the naming of the adapters is significant, and the order of the adapters made a difference. Anyway, I only needed a single /etc/sysconfig/network-scripts file, for the enp0s3 network (this actually already existed):

TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp0s3
DEVICE=enp0s3
ONBOOT=yes
DNS1=8.8.8.8
DNS2=8.8.4.4
IPADDR=192.168.63.101
NETMASK=255.255.255.0

It may also be necessary to fiddle with /etc/resolv.conf, however as the NetworkManager application is running, this seems to be correctly getting the nominated DNS servers into play:

# Generated by NetworkManager
search xx.xxxxx.com
nameserver 153.65.20.113
nameserver 8.8.8.8
nameserver 8.8.4.4

The IP address and gateway come from the virtual network set up by VirtualBox. If I have a look on my desktop, I can see (some details omitted):

$ ifconfig
vboxnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
     ether 0a:00:27:00:00:00
inet 192.168.63.1 netmask 0xffffff00 broadcast 192.168.63.255
vboxnet1: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
     ether 0a:00:27:00:00:01

revealing the IP range used by this network (this can also be found buried in the preferences for VirtualBox). And there we have it – a fairly minimal Centos/7 box onto which I can throw NiFi, and start to play:

$ ssh centos@192.168.63.101
centos@192.168.63.101's password:
Last login: Thu Jun 29 11:00:59 2017 from gateway

[centos@localhost ~]$ sudo -s
[root@localhost centos]# nslookup nifi.apache.org
Server:  153.65.20.113
Address: 153.65.20.113#53
Non-authoritative answer:
Name: nifi.apache.org
Address: 88.198.26.2
Name: nifi.apache.org
Address: 140.211.11.105

[root@localhost centos]# java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) Client VM (build 25.131-b11, mixed mode)
[root@localhost centos]#        

Installing NiFi

To begin with, I created a user nifi, and installed NiFi:

$ ssh centos@192.168.63.101
centos@192.168.63.101's password:
[centos@localhost ~]$ sudo adduser nifi
[centos@localhost ~]$ sudo yum install wget
[centos@localhost ~]$ sudo -i -u nifi
[nifi@localhost ~]$ wget http://apache.spinellicreations.com/nifi/1.3.0/nifi-1.3.0-bin.tar.gz
[nifi@localhost ~]$ tar xvf nifi-1.3.0-bin.tar.gz
[nifi@localhost ~]$ ln -s nifi-1.3.0 ./latest

Since I would like this to be running as a service when the VM boots, I went ahead and created /etc/systemd/system/nifi.service as root:

[Unit]
Description=Apache NiFi
After=network.target

[Service]
Type=forking
User=nifi
Group=nifi
ExecStart=/home/nifi/latest/bin/nifi.sh start
ExecStop=/home/nifi/latest/bin/nifi.sh stop
ExecReload=/home/nifi/latest/bin/nifi.sh restart

[Install]
WantedBy=multi-user.target

Note that I’ve created the nifi user without a password, so we cannot login as that user, however we will run the service as nifi rather than root. Having created the service definition, we need to enable it:

[centos@localhost ~]$ sudo systemctl daemon-reload
[centos@localhost ~]$ sudo systemctl enable nifi

That’s not the end though – the minimal ISO that I based this on fortuitously has almost every port locked off behind a firewall, so we need to crack port 8080 open:

[centos@localhost ~]$ sudo firewall-cmd --zone=public --permanent --add-port=8080/tcp
[centos@localhost ~]$ sudo firewall-cmd --reload

At this point, I was able to successfully access http://192.168.63.101:8080/nifi from my laptop, so I saved the snapshot and was ready to start cloning it.

I created three clones, one at a time, and for each of them updated /etc/sysconfig/network-scripts/ifcfg-enp0s3 to set a distinct IP address. I also modified the hostname for convenience

[root@localhost centos]# hostnamectl set-hostname node1.localdomain
[root@localhost centos]# hostnamectl status
     Static hostname: node1.localdomain
                 Icon name: computer-vm
                     Chassis: vm
                Machine ID: 894ba5698635417bbf95d888b886cfb2
                     Boot ID: 7c45df1521184636986f40976044b2ee
        Virtualization: kvm
    Operating System: CentOS Linux 7 (Core)
             CPE OS Name: cpe:/o:centos:centos:7
                        Kernel: Linux 3.10.0-514.21.2.el7.x86_64
            Architecture: x86-64

and added the following to /etc/hosts in each machine, and on my local host:

192.168.63.102  node1.localdomain node1
192.168.63.103  node2.localdomain node2
192.168.63.104  node3.localdomain node3

At this stage, I can:

• run all three VMs at the same time
• each VM can ping each other
• login to each of the VMs from my desktop
• access the (distinct) NiFi instances on each node from my desktop at http://node[123]:8080/nifi

Next stage will be to build these into a cluster!

Setting up a NiFi Cluster

To begin with, some extra ports need to be opened up in the firewall (executed as root):

$ firewall-cmd --zone=public --permanent --add-port=2181/tcp
$ firewall-cmd --zone=public --permanent --add-port=2888/tcp
$ firewall-cmd --zone=public --permanent --add-port=3888/tcp
$ firewall-cmd --zone=public --permanent --add-port=9998/tcp
$ firewall-cmd --zone=public --permanent --add-port=9999/tcp
$ firewall-cmd --reload
$ firewall-cmd --list-all

The final list command should reveal which ports have been cracked open:

[centos@node3 ~]$ sudo firewall-cmd --list-all
public (active)
    target: default
    icmp-block-inversion: no
    interfaces: enp0s3 enp0s8
    sources:
    services: dhcpv6-client ssh
    ports: 3888/tcp 2181/tcp 8080/tcp 2888/tcp 9999/tcp 9998/tcp
    protocols:
    masquerade: no
    forward-ports:
    sourceports:
    icmp-blocks:
    rich rules:

As an aside, the tool that really helped me to diagnose what was going on was nmap, which I can highly recommend as a sledgehammer for unambiguously testing whether you can reach a particular box on a particular port.

There are three NiFi configuration files to be modified, all in ~nifi/latest/conf:

zookeeper.properties

server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888

state-management.xml

<cluster -provider>
    <id>zk-provider</id>
    <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
    <property name="Connect String">node1:2181,node2:2181,node3:2181</property>
    <property name="Root Node">/nifi</property>
    <property name="Session Timeout">10 seconds</property>
    <property name="Access Control">Open</property>
</cluster>

nifi.properties

note that nifi.properties on each node is slightly different, in that it contains the node name (here node3):

nifi.cluster.flow.election.max.wait.time=2 mins
nifi.cluster.flow.election.max.candidates=3
nifi.state.management.embedded.zookeeper.start=true
nifi.remote.input.host=node3
nifi.remote.input.socket.port=9998
nifi.web.http.host=node3
nifi.cluster.is.node=true
nifi.cluster.node.address=node3
nifi.cluster.node.protocol.port=9999
nifi.zookeeper.connect.string=node1:2181,node2:2181,node3:2181

By specifying the max.candidates value to be equal to the number of nodes, then we avoid having a lengthy wait for the election of a master node during startup – as soon as each node sees the other two nodes, they will settle one of themselves to be the master.

Bringing this cluster up, you will be able to observe in the logs of each of the NiFi instance that each node is sending a heartbeat out every few seconds, and similarly you should see whichever node is currently the master handling the heartbeats and taking responsibility for preserving state.
You will also be able to observe that if you connect to the UI of one node in the cluster and make changes to the processors, the changes will be echoed to all the other nodes – NiFi clustering uses a zero-master configuration, so as long as the cluster is healthy, there is no privileged instance of the UI.

So there we have it – a three-node NiFi cluster on the desktop, using minimal Centos/7 virtual machines in VirtualBox