This guide contains everything you need to get a basic Hadoop cluster up and running. It is intended as a condensed and easy-to-understand supplement to the official documentation, so lengthy descriptions are omitted. For the full documentation, see Cloudera's Install Guide.

Whether you want to start with a basic two-node cluster, or add hundreds or even thousands of nodes, the concepts here apply. Adding nodes can be done at any time without interrupting the cluster's workflow, so as long as you have two machines, you're ready to begin installing.

About hardware:

Hadoop was designed to be used on commodity hardware, so you won't need anything special for this project. You'll want 2 or more reasonably fast, modern servers. (I'm using 9 SuperMicro boxes I happened to have lying around). Mine have dual Xeon processors, 48G RAM, 4 x 7200 RPM SATA disks in JBOD mode. JBOD is recomended above RAID for Hadoop, since Hadoop has its own built-in redundancy which performs better with plain disks.

Unlike High-Availablilty clustering, an HPC cluster like Hadoop does not require any special fencing hardware. It handles hardware failure by simply not giving jobs to broken/misbehaving nodes. If a node fails a certain number of jobs, it's out.

In a Hadoop cluster, the only type of hardware failure that would cause any noticeable disruption is the possible failure of the NameNode. This is why it's always good to have a Secondary NameNode on standby.

Hadoop Clusters - Core concepts

Here are a few core concepts that will help you understand what you're about to build. For a very small cluster (let's say, 9 nodes or less), your cluster will consist of these types of nodes:

  1. Head node. Runs the NameNode service and JobTracker service.
  2. Worker nodes. All other nodes in the cluster will run DataNode and TaskTracker services.
A larger cluster is almost identical to this, but generally they use a separate machine for the JobTracker service. It's also common to add a Secondary NameNode for redundancy. So in that scenerio you'd have:
  1. NameNode machine.
  2. Secondary NameNode machine.
  3. Jobtracker machine.
  4. Worker machines, each running DataNode + TaskTracker services.
A brief definition of these components:
  • NameNode: Stores all metadata for the HDFS filesystem. 
  • DataNodes:  Worker nodes that store and retrieve data when told to (by clients or the NameNode).
  • TaskTrackers: Runs tasks and sends progress reports to the JobTracker.
  • JobTracker: Coordinates all jobs and schedules them to run on TaskTracker nodes.
  • HDFS: Hadoop Distributed File System. An HDFS cluster consists of a NameNode + DataNodes. All Hadoop IO happens through this. Built for storing very large files across many machines.

Hadoop Installation 

Now that you have a little background on this software, we can begin installing. The first thing you'll need is Java JDK 1.6 u8 or higher. You might also want to use a tool like clusterssh to ssh into all your nodes at once to perform this installation.

 Install Java on each node

 # grab the latest java version - probably not this one anymore ;)
rpm -Uvh jdk-7u1-linux-x64.rpm
alternatives --install /usr/bin/java java /usr/java/latest/bin/java 1600
alternatives --auto java

Disable SELinux

setenforce 0
vim /etc/sysconfig/selinux

Allow communication between nodes in IPtables

Either disable IPtables, allow all communication between nodes, or open Hadoop-specific ports.

Set up the Cloudera yum repo

yum --nogpgcheck localinstall cdh3-repository-1.0-1.noarch.rpm
rpm --import 

On the head node, install NameNode and JobTracker packages

yum -y install hadoop-0.20-namenode hadoop-0.20-jobtracker

On the worker nodes, install DataNode and TaskTracker packages

yum -y install hadoop-0.20-datanode hadoop-0.20-tasktracker

Use alternatives to set up your custom cluster config

Setting up your cluster like this will allow you to keep multiple cluster configurations handy, and makes switching between them so simple!

cp -r /etc/hadoop-0.20/conf.empty /etc/hadoop-0.20/conf.MyCluster
alternatives --install /etc/hadoop-0.20/conf hadoop-0.20-conf /etc/hadoop-0.20/conf.MyCluster 50 
alternatives --set hadoop-0.20-conf /etc/hadoop-0.20/conf.MyCluster

Verify that it's using MyCluster config instead of the default

[root@nodes ~]# alternatives --display hadoop-0.20-conf 
hadoop-0.20-conf - status is auto.
link currently points to /etc/hadoop-0.20/conf.MyCluster
/etc/hadoop-0.20/conf.empty - priority 10
/etc/hadoop-0.20/conf.MyCluster - priority 50
Current `best' version is /etc/hadoop-0.20/conf.MyCluster.

Set up Hadoop storage disks on each node

In a Hadoop cluster, it is ideal to have multiple plain-disk storage mounts on each node. In this example, I'm using 4 plain disks, formatted ext4.

mkdir -p /mnt/hdfs/{1..4}

Add the new disks to /etc/fstab, ensuring that they're mounted with noatime. (This prevents reads from turning into unnecessary writes, which is generally good for performance.)

vim /etc/fstab
# append the new disks
/dev/sdb1 /mnt/hdfs/1 ext4 noatime 0 0
/dev/sdc1 /mnt/hdfs/2 ext4 noatime 0 0
/dev/sdd1 /mnt/hdfs/3 ext4 noatime 0 0
/dev/sde1 /mnt/hdfs/4 ext4 noatime 0 0

Mount the new disks and create Hadoop directories

The directories we're creating here correspond with Hadoop configuration options we'll be setting later:

mount /mnt/hdfs/1
mount /mnt/hdfs/2
mount /mnt/hdfs/3
mount /mnt/hdfs/4

# create the namenode, datanode, and mapred dirs on each disk
for num in 1 2 3 4; do mkdir /mnt/hdfs/$num/{namenode,datanode,mapred}; done

Set directory permissions

This part is very important! You'll run into errors later if these dirs are owned by the wrong user. 

# make sure everything is owned by hdfs:hadoop  
chown -R hdfs:hadoop /mnt/hdfs/
# ...except for the mapred dirs 
chown -R mapred:hadoop /mnt/hdfs/{1,2,3,4}/mapred

Hadoop Core Configuration

Now that the underlying directories and storage devices are set up, we're ready to configure Hadoop.

These configuration files are not node-specific, so you can write them once and copy them to all nodes. Change these config examples so that HEADNODE is actually the name of your head node.

This config file will tell Hadoop where to find the NameNode and its default file system.


This is where we tell Hadoop to use the directories we created earlier. It specifies local storage on each node, used by the DataNodes and NameNode services to store HDFS data. 


Specify the JobTracker here, along with all the local directories for writing map/reduce (job-related) data. This is used by the TaskTracker service on all the worker nodes. Change HEADNODE to the name of your machine that runs the JobTracker service. (In a small cluster, this machine is the same one that runs the NameNode service.)



Bringing the Cluster online

With the basic configuration finished above, the cluster is now ready to be brought online. First, we need to format the NameNode to create an HDFS filesystem for our nodes to use as storage. This only needs to be done on the NameNode.

Formatting the NameNode

[root@HEADNODE ~]# sudo -u hdfs hadoop namenode -format

11/12/07 04:55:59 INFO namenode.NameNode: STARTUP_MSG:
STARTUP_MSG: Starting NameNode
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.2-cdh3u2
STARTUP_MSG: build = file:///tmp/topdir/BUILD/hadoop-0.20.2-cdh3u2 -r 95a824e4005b2a94fe1c11f1ef9db4c672ba43cb; compiled by 'root' on Thu Oct 13 21:51:01 PDT 2011
Format filesystem in /mnt/hdfs/1/namenode ? (Y or N) Y
Format filesystem in /mnt/hdfs/2/namenode ? (Y or N) Y
Format filesystem in /mnt/hdfs/3/namenode ? (Y or N) Y
Format filesystem in /mnt/hdfs/4/namenode ? (Y or N) Y
11/12/07 04:56:00 INFO util.GSet: VM type = 64-bit
11/12/07 04:56:00 INFO util.GSet: 2% max memory = 17.77875 MB
11/12/07 04:56:00 INFO util.GSet: capacity = 2^21 = 2097152 entries
11/12/07 04:56:00 INFO util.GSet: recommended=2097152, actual=2097152
11/12/07 04:56:00 INFO namenode.FSNamesystem: fsOwner=hdfs
11/12/07 04:56:00 INFO namenode.FSNamesystem: supergroup=supergroup
11/12/07 04:56:00 INFO namenode.FSNamesystem: isPermissionEnabled=true
11/12/07 04:56:00 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=1000
11/12/07 04:56:00 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
11/12/07 04:56:00 INFO common.Storage: Image file of size 110 saved in 0 seconds.
11/12/07 04:56:01 INFO common.Storage: Storage directory /mnt/hdfs/1/namenode has been successfully formatted.
11/12/07 04:56:01 INFO common.Storage: Storage directory /mnt/hdfs/2/namenode has been successfully formatted.
11/12/07 04:56:01 INFO common.Storage: Storage directory /mnt/hdfs/3/namenode has been successfully formatted.
11/12/07 04:56:01 INFO common.Storage: Storage directory /mnt/hdfs/4/namenode has been successfully formatted.
11/12/07 04:56:01 INFO namenode.NameNode: SHUTDOWN_MSG:
SHUTDOWN_MSG: Shutting down NameNode at HEADNODE/

We can see from the output above that the NameNode has been successfully formatted. But nothing is running yet, so let's start up our NameNode service on the head node. To start up services, first we'll need to fix permissions on each node.

HDFS User Permissions

Set up permissions to allow the hdfs user to write log files here. Otherwise, the service may fail to start correctly.

chgrp hdfs /usr/lib/hadoop-0.20/ 
chmod g+rw /usr/lib/hadoop-0.20/

Start the NameNode service on the head node

service hadoop-0.20-namenode start

And on the worker nodes, start up the DataNode service.

service hadoop-0.20-datanode start

Give it a couple seconds to start up the HDFS filesystem. The nodes will connect, and the local storage of each node will be added to the collective HDFS filesystem. Now we can create core directories.

First-time HDFS use: create core directories

# these paths are relative to the HDFS filesystem, 
# so you can copy and paste this regardless of your physical directory layout
sudo -u hdfs hadoop fs -mkdir /mapred/system
sudo -u hdfs hadoop fs -chown mapred:hadoop /mapred/system
sudo -u hdfs hadoop dfs -mkdir /tmp
sudo -u hdfs hadoop dfs -chmod -R 1777 /tmp

Check that your nodes are all online and healthy

This will show a report overview for all nodes, and show you how much storage you have between them all.

sudo -u hdfs hadoop dfsadmin -report
sudo -u hdfs hadoop dfs -df

Start JobTracker and TaskTracker services

If all looks well, continue on to start the JobTracker service on the head node.

service hadoop-0.20-jobtracker start

And start the TaskTracker service on the worker nodes.

service hadoop-0.20-tasktracker start

You now have a fully-functional Hadoop cluster up and running! Check the cluster status on your local Hadoop status pages:




Category: Blog