application melding

making square pegs fit in round holes

Apache Hadoop + Docker + Fedora: Running Images

In a previous post I discussed how to generate docker images that include a pre-configured and simple hadoop setup. Now it’s time to run them and that provides us with our first hurdle: DNS

Hadoop and DNS

If you’ve ever tried to run a hadoop cluster without DNS you may not have gotten very far. The namenode appears to use DNS lookups to verify datanodes that try to connect to it. If DNS isn’t setup properly then the namenode will never show any datanodes connected to it and if you check the namenode logs you’ll see an odd error like:

error: DisallowedDatanodeException: Datanode denied communication with namenode: DatanodeRegistration(<ip|hostname>, storageID=DS-1141974467-172.17.0.14-50010-1393006490185, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-555184a7-6958-41d3-96d2-d8bcc7211819;nsid=1734635015;c=0)

Using only IP addresses in the configuration files won’t solve the problem either, as the namenode appears to do a reverse lookup for nodes that come in with an IP instead of a hostname. Where’s the trust?

There are a few solutions to this depending on the version of hadoop that will be installed in the image. Fedora 20 has hadoop 2.2.0 and will require a DNS solution which I’ll detail in just a bit. I’m working on updating hadoop to 2.4.0 for Fedora 21 and that has a configuration option that was introduced in 2.3.0 that may allow you to disable reverse DNS lookups from the namenode when datanodes register. For hadoop 2.3.0 and beyond you may avoid the need to set up a DNS server by adding the following snipet to the hdfs-site.xml config file:

<property>
    <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
    <value>false</value>
</property>

If you need to setup a DNS server it isn’t too hard, but it does limit how functional these containers can be since the hostname of the datanodes will need to be resolvable in DNS. That’s not too bad for a local setup where IP addresses can be easily controlled, but when you branch out to using multiple physical hosts this can be a problem for a number of reasons. I’ll go through some of the limitations in a later post, but for now I’ll go through using dnsmasq to setup a local DNS server to get these containers functioning on a single host.

Setting Up Dnsmasq

This is pretty well covered in the README, but I’ll cover it again here in a little more detail. First we need to install dnsmasq:

yum install dnsmasq

Next you’ll need to configure dnsmasq to listen on the bridge interface docker will use. By default that is the interface docker0. To tell dnsmasq to listen on that interface:

echo "interface=docker0" >> /etc/dnsmasq.conf

Next we need to setup the forward and reverse resolvers. Create a 0hosts file in /etc/dnsmasq.d and add these entries to it:

address="/namenode/<namenode_ip>"
ptr-record=<reverse_namenode_ip>.in-addr.arpa,namenode
address="/resourcemgr/<resourcemgr_ip>"
ptr-record=<reserve_resourcemgr_ip>.in-addr.arpa,resourcemgr
address="/datanode1/<datanode_ip>"
ptr-record=<reverse_datanode_ip>.in-addr.arpa,datanode1

The hostnames for the namenode and resource manager are important if using images generated from the Dockerfiles I pointed at earlier.

What IP addresses should you use? Well, that’s a slightly more complicated answer than it seems because of how docker hands out IP addresses. I’m going to use an example where the namenode is given the IP address 172.17.0.2, so the DNS entries for the namenode with that IP address is:

address="/namenode/172.17.0.2"
ptr-record=2.0.17.172.in-addr.arpa,namenode

If you want to add more datanodes to the pool you’ll obviously need to add more entries to the DNS records. Now that we’ve got dnsmasq configured let’s start it:

systemctl start dnsmasq

Starting the Containers

Now that we have DNS setup we can start some containers. As you might expect there’s a catch here as well. The containers must be started in the following order:

  1. namenode
  2. resource manager
  3. datanode(s)

This startup order is imposed by the hadoop daemons and what they do when they fail to contact another daemon they depend upon. In some instances I’ve seen the daemons attempt to reconnect, and others I’ve seen them just exit. The surefire way to get everything up and running is to start the containers in the order I provided.

To start the namenode, execute:

docker run -d -h namenode --dns <dns_ip> -p 50070:50070 <username>/hadoop-namenode

What this command is doing is important so I’ll break it down piece by piece:

  • -d: Run in daemon mode
  • -h: Give the container this hostname
  • —dns: Set this as the DNS server in the container. It should be the IP address of the router inside docker. This should always be the first IP address in the subnet determined by the bridge interface.
  • -p: Map a port on the host machine to a port on the container

For the containers and the DNS setup I’ve been detailing in my posts using the default docker bridge interface I would execute:

docker run -d -h namenode --dns 172.17.0.1 -p 50070:50070 rrati/hadoop-namenode

The resource manager and the datanode are started similarly:

docker run -d -h resourcemgr --dns <dns_ip> -p 8088:8088 -p 8032:8032 <username>/hadoop-resourcemgr
docker run -d -h datanode1 --dns <dns_ip> -p 50075:50075 -p 8042:8042 <username>/hadoop-datanode

Make sure that the hostnames provided with the -h option match the hostnames you setup in dnsmasq.

Using External Storage

This setup is using HDFS for storage, but that’s not going to do us much good if the everything in the namenode or a datanode is lost every time a container is stopped. To get around that you can map directories into the container on startup. This would allow the container to write data to a location that won’t be destroyed when the container is shut down. To map a directory into the namenode’s main storage location, you would execute:

docker run -d -h namenode —dns 172.17.0.1 -p 50070:50070 -v <persistent_storage_dir>:/var/cache/hadoop-hdfs rrati/hadoop-namenode

This will mount whatever directory pointed to by <persistent_storage_dir> in the container at /var/lib/hadoop-hdfs. The storage directory will need to be writable by the user running the daemon in the container. In the case of the namenode, the daemon is run by the user hdfs.

Submitting a Job

We’re about ready to submit a job into the docker cluster we started. First we need to setup our host machine to talk to the hadoop cluster. This is pretty simple and there are a few ways to do it. Since I didn’t expose the appropriate ports when I started the namenode and resourcemanager I will use the hostnames/IPs of the running containers. I could have exposed the required ports when I started the containers and pointed the hadoop configuration files at localhost:, but for this example I elected not to do that.

First you need to install some hadoop pieces on the host system:

yum install hadoop-common hadoop-yarn

Then you’ll need to modify /etc/hadoop/core-site.xml to point at the namenode. Replace the exist property definition for the follwing with:

<property>
  <name>fs.default.name</name>
  <value>hdfs://namenode:8020</value>
</property>

For simplity I use the hostnames I setup in my DNS server so I only have one location I have to deal with if IPs change. You also have to make sure to add the dnsmasq server to the list of DNS servers in /etc/resolv.conf if you do it this way. Using straight IPs works fine as well.

Next you’ll need to add this to /etc/hadoop/yarn-site.xml:

<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>resourcemgr</value>
</property>

Again I’m using the hostname defined in the dnsmasq server. Once you make those two changes you can submit a job to your hadoop cluster running in your containers.