Contribute Limited Amount Of Storage Of DataNode In Hadoop Cluster
đź”° TASK DESCRIPTION:
🤔 In Hadoop cluster , find how to contribute limited/specific amount of storage as a slave node to the cluster ?
Hint : Use Of Partitions
Step 1: Add New HardDisk to DataNode
- I am using Oracle Virtual Box so we don’t need to purchase new hard disk . We will use Virtual Hard Disk concept.
To add new hard disk DataNode must be in “Stopped” state then Follow this steps :-
(A) Open Storage in Settings of DataNode -
(B) Click on “Controller: SATA” & after this click on right “+” icon of “Controller: SATA” :-
© Click On “Create” :-
(D) Then click Next-Next and select the size of virtual disk you want to attach.
- In my case My Hard Disk size is 20 GiB.
(E) To check hard disk is attached or not run “fdisk -l” command :-
Step 2: Create Partition In virtual Disk 0f DataNode :-
Run this command :-
fdisk /dev/sdb
I want to create 5 GiB partition so that DataNode can only use 5GiB for contribution in Hadoop Cluster.
- Run this command to check partition :-
fdisk -l /dev/sdb
Run this command to load Driver for Partition :-
udevadm settle
- Whenever New device is added in Computer then we have to load respectively driver so that we can communicate with that device.
Step 3: Format & Mount Partition at DataNode
(A) Run this command to format :-
mkfs.ext4 /dev/sdb1
- In my case I am using “ext4” format type ,you can choose according to you.
(B) Run this command to mount :-
mount /dev/sdb1 /data1
Step 4: Configure NameNode :-
(A) Make a Directory :-
mkdir /namenode
(B) Format the Directory :-
hadoop namenode -format
(C) “hdfs-site.xml” file configuration :-
(D) “core-site.xml” file configuration :-
(E) Starting NameNode :-
hadoop-daemon.sh start namenode
Step 5: Configure DataNode :-
(A) “hdfs-site.xml” file configuration
(B) “core-site.xml” file configuration :-
(C) Starting DataNode :-
hadoop-daemon.sh start datanode
Step 6: Check Contribution Of DataNode In Distributed File Storage of Hadoop Cluster :-
hadoop dfsadmin -report
You can see DataNode is contributing around 5GiB . Thus we can set limitation of contribution of DataNode in Hadoop Cluster.