Contribute Limited Amount Of Storage Of DataNode In Hadoop Cluster

🔰 TASK DESCRIPTION:

🤔 In Hadoop cluster , find how to contribute limited/specific amount of storage as a slave node to the cluster ?

Hint : Use Of Partitions

Step 1: Add New HardDisk to DataNode

• I am using Oracle Virtual Box so we don’t need to purchase new hard disk . We will use Virtual Hard Disk concept.

To add new hard disk DataNode must be in “Stopped” state then Follow this steps :-

(A) Open Storage in Settings of DataNode -

(B) Click on “Controller: SATA” & after this click on right “+” icon of “Controller: SATA” :-

© Click On “Create” :-

(D) Then click Next-Next and select the size of virtual disk you want to attach.

• In my case My Hard Disk size is 20 GiB.

(E) To check hard disk is attached or not run “fdisk -l” command :-

Step 2: Create Partition In virtual Disk 0f DataNode :-

Run this command :-

fdisk /dev/sdb

I want to create 5 GiB partition so that DataNode can only use 5GiB for contribution in Hadoop Cluster.

• Run this command to check partition :-

fdisk -l /dev/sdb

Run this command to load Driver for Partition :-

udevadm settle

• Whenever New device is added in Computer then we have to load respectively driver so that we can communicate with that device.

Step 3: Format & Mount Partition at DataNode

(A) Run this command to format :-

mkfs.ext4 /dev/sdb1

• In my case I am using “ext4” format type ,you can choose according to you.

(B) Run this command to mount :-

mount /dev/sdb1 /data1

Step 4: Configure NameNode :-

(A) Make a Directory :-

mkdir /namenode

(B) Format the Directory :-

hadoop namenode -format

(C) “hdfs-site.xml” file configuration :-

(D) “core-site.xml” file configuration :-

(E) Starting NameNode :-

hadoop-daemon.sh start namenode

Step 5: Configure DataNode :-

(A) “hdfs-site.xml” file configuration

(B) “core-site.xml” file configuration :-

(C) Starting DataNode :-

hadoop-daemon.sh start datanode

Step 6: Check Contribution Of DataNode In Distributed File Storage of Hadoop Cluster :-

hadoop dfsadmin -report

You can see DataNode is contributing around 5GiB . Thus we can set limitation of contribution of DataNode in Hadoop Cluster.