Technology ·

Hadoop Bullet: a simple script to deploy Hadoop on fresh machine in automated fashion

Installing Hadoop is a hassle; it involves a variety of steps, some proficiency on Linux commands and writing to various files. If you have tried manual installation, you know what I'm talking about.

So, here is a simple Linux shell script. Save the following script as bulletinstall.sh and on your Ubuntu-ready machine, run it using:

$ sudo sh bulletinstall.sh

This script has been tested on Ubuntu 14.04 LTS; if you experience any issues, feel free to drop a comment. Here is the script:

#!/bin/bash

# This document is free to share and/or modify, and comes with ABSOLUTELY NO WARRANTIES. I will not be responsible for any damage or corruption caused to your Computer. Do know your stuff before you run this and backup your important files before trying out.
# Author: owaishussain@outlook.com
# LINUX SCRIPT TO INSTALL HADOOP 1.2.1 ON A MACHINE
# If you already have this file, then put it in /tmp directory and comment out "wget"
HADOOP_URL=http://www.us.apache.org/dist/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz

echo "************************************************************"

cd /tmp

echo "Downloading Apache Hadoop from $HADOOP_URL (you may change the version to any other, but this one has been tested)"

wget "$HADOOP_URL"

echo "STEP 1/10 COMPLETE..."

echo "************************************************************"

# Hadoop will be deployed on "/usr/local" directory, if you want to change it, then modify the path under this section
echo "************************************************************"
echo "Extracting files..."
tar -xzf hadoop-1.2.1.tar.gz

echo "Copying to \"/usr/local/\""

cp -R hadoop-1.2.1 /usr/local/

echo "STEP 2/10 COMPLETE..."

echo "************************************************************"

# Oracle JDK7 will be downloaded from a 3rd party repository (ppa:webupd8team/java. Credit to them). In case, it is unavailable, you may use an alternative in the first command under this section

echo "************************************************************"

echo "Installing Oracle JDK7. Please accept the Oracle license when asked"

add-apt-repository ppa:webupd8team/java

apt-get update

# If you are skeptik about JDK7 (and I won't blame you for that), you may switch comments respectively, in the two lines below to install JDK6

apt-get install oracle-java7-installer

#apt-get install sun-java6-jdk

echo "STEP 3/10 COMPLETE..."

echo "************************************************************"

# This script assumes that you will run Hadoop on new user \"hadoop\". If you wish to choose different user, you're most welcome to; just comment whole section below, but be sure to replace the username everywhere in the script with what you want
echo "************************************************************"
echo "Creating user and group named\" hadoop\""
adduser hadoop
adduser hadoop sudo
echo "Creating home directory for user"
mkdir -p /home/hadoop/tmp
echo "Assigning rights on home directory"
chown -R hadoop:hadoop /home/hadoop/tmp
chmod 755 /home/hadoop/tmp
echo "Changing ownership of Hadoop's installation directory"
chown -R hadoop:hadoop /usr/local/hadoop-1.2.1

echo "STEP 4/10 COMPLETE..."

echo "************************************************************"

# This is to test and initialize the newly created user

echo "************************************************************"

echo "Logging in. ***** PLEASE PROVIDE PASSWORD AND RUN 'exit' (without quotes) TO LOG OUT *****"

su - hadoop

echo "STEP 5/10 COMPLETE..."

echo "************************************************************"

echo "************************************************************"

echo "Setting environment variables"

cd /usr/local/hadoop-1.2.1/conf

echo "export JAVA_HOME=/usr/lib/jvm/java-7-oracle" >> hadoop-env.sh

echo "STEP 6/10 COMPLETE..."

echo "************************************************************"

echo "************************************************************"

echo "Configuring Hadoop properties in *site.xml"

mv core-site.xml core-site.xml.bck

touch core-site.xml

echo "<?xml version=\"1.0\"?>

<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
</property>
</configuration>" > core-site.xml
mv mapred-site.xml mapred-site.xml.bck
touch mapred-site.xml
echo "<?xml version=\"1.0\"?>
<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
</property>
</configuration>" > mapred-site.xml
mv hdfs-site.xml hdfs-site.xml.bck
touch hdfs-site.xml
echo "<?xml version=\"1.0\"?>
<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>" > hdfs-site.xml
chown hadoop:hadoop /usr/local/hadoop-1.2.1/conf/*-site.xml

echo "STEP 7/10 COMPLETE..."

echo "************************************************************"

echo "************************************************************"

echo "export JAVA_HOME=/usr/lib/jvm/java-7-oracle" >> /home/hadoop/.bashrc

echo "export HADOOP_HOME=/usr/local/hadoop-1.2.1" >> /home/hadoop/.bashrc

echo "export PATH=$PATH:/usr/local/hadoop-1.2.1/bin" >> /home/hadoop/.bashrc

echo "STEP 8/10 COMPLETE..."

echo "************************************************************"

echo "************************************************************"

echo "Installing OpenSSH Server"

apt-get install openssh-server

ssh-keygen -t rsa -P ""
cat /home/hadoop/.ssh/id_rsa.pub >> /home/hadoop/.ssh/authorized_keys

echo "STEP 9/10 COMPLETE..."

echo "************************************************************"

echo "************************************************************"

echo "Removing temporary files"

# If you want to preserve the downloaded Hadoop application, you may comment out the first command under this section

rm hadoop-1.2.1.tar.gz

rm -R hadoop-1.2.1

echo "STEP 10/10 COMPLETE..."

echo "************************************************************"

echo "Congratulations! Your Hadoop setup is complete. Please log into hadoop user and start hadoop services."

echo "************************************************************"