How to install Hadoop on Mac

Step 1: Install Homebrew

$ ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

How to fix homebrew permissions?

i have uninstalled and installed Homebrew 3 times now because it seems to never allow me to install anything as it denies me permissions at the end of most installations.

As an example i wil post this libjpeg download scenario that im currently facing.

i try to install libjpeg and get

$ brew install libjpeg
==> Downloading https://downloads.sf.net/project/machomebrew/Bottles/jpeg-8d.mountain_lion.bottle.1.tar.gz
Already downloaded: /Library/Caches/Homebrew/jpeg-8d.mountain_lion.bottle.1.tar.gz
==> Pouring jpeg-8d.mountain_lion.bottle.1.tar.gz
Warning: Could not link jpeg. Unlinking...
Error: The brew link step did not complete successfully
The formula built, but is not symlinked into /usr/local
You can try again using `brew link jpeg'
Error: Permission denied - /usr/local/opt/jpeg

‘brew link jpeg’ results in

Error: Permission denied - /usr/local/opt/jpeg

Here is what my brew doctor reads

$ brew doctor
Warning: "config" scripts exist outside your system or Homebrew directories.
./configure scripts often look for *-config scripts to determine if
software packages are installed, and what additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew provided
script of the same name. We found the following "config" scripts:

/Library/Frameworks/Python.framework/Versions/2.7/bin/python-config
/Library/Frameworks/Python.framework/Versions/2.7/bin/python2-config
/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-config
Warning: You have unlinked kegs in your Cellar
Leaving kegs unlinked can lead to build-trouble and

I was able to solve the problem by using chrown on the folder like so

sudo chown -R $USER:admin /usr/local

Apparently i had used sudo before in a way that altered my folder permission on /usr/local, from here on forward all installations with brew have proven to be successful.

This answer comes courtesy of gitHub’s homebrew issue tracker

 

Step 2: Install Hadoop

$ brew install hadoop
Let’s assume that brew installs Hadoop 1.1.2.

Step 3: Configure Hadoop

$ cd /usr/local/Cellar/hadoop/1.1.2/libexec

Add the following line to conf/hadoop-env.sh:

1
export HADOOP_OPTS="-Djava.security.krb5.realm= -Djava.security.krb5.kdc="

Add the following lines to conf/core-site.xml inside the configuration tags:

1
2
3
4
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>

Add the following lines to conf/hdfs-site.xml inside the configuration tags:

1
2
3
4
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>

Add the following lines to conf/mapred-site.xml inside the configuration tags:

1
2
3
4
<property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
</property>

Step 4: Enable SSH to localhost

Go to System Preferences > Sharing.
Make sure “Remote Login” is checked.
$ ssh-keygen -t rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Step 5: Format Hadoop filesystem

$ bin/hadoop namenode -format

Step 6: Start Hadoop

$ bin/start-all.sh

Make sure that all Hadoop processes are running:
$ jps

Run a Hadoop example:
$ bin/hadoop jar /usr/local/Cellar/hadoop/1.1.2/libexec/hadoop-examples-1.1.2.jar pi 10 100

Hadoop logs: /usr/local/Cellar/hadoop/1.1.2/libexec/logs/
Web interface for Hadoop NameNode: http://localhost:50070/
Web interface for Hadoop JobTracker: http://localhost:50030/

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s