Adding more space to your Cloudera Hadoop nodes

I noticed that my Cloudera cluster install seemed to put all the DFS directories on the root volume of each node at /dfs/dn. There were 2 volumes on each machine and the larger volume was mounted at /home. I can’t recall why or if I had something to do with that. However, needless to say, most of the space on this cluster was not even being used by the cluster in any way. Here’s how I got my Hadoop install to use the extra space:

Note you will have to restart your cluster; or if you have Enterprise, do a rolling restart.

On each node, create a directory at /home/dfs/dn. It’s in /home *only* because I didn’t want to redo all the mount points and change up partitions. I plan on nuking each data node one at a time and installing more disk space anyways, so this will do for now. Anyways…

1. create the directory you want HDFS to use

– mkdir /home/dfs

– mkdir /home/dfs/dn

– chown -R hdfs:hdfs /home dfs

2. Go to your Cloudera Manager web interface and click on “Home” at the top > HDFS service in the relevant Cluster > Instances > “DataNode” (you’ll do this for each node) > Configuration > now click the “+” sign in the DataNode Data Directory config section and type in “/home/dfs/dn”.

3. Go back to “Home” and you’ll see an icon next to the HDFS service that indicates restarts are necessary. Do that.

Once the cluster comes back up, new data writes should start going to the new directories. You should also see the bar in the HDFS Summary area indicate the additional available space.