Contribute Limited Amount Of Storage Of DataNode In Hadoop Cluster


🤔 In Hadoop cluster , find how to contribute limited/specific amount of storage as a slave node to the cluster ?

Hint : Use Of Partitions

Step 1: Add New HardDisk to DataNode

  • I am using Oracle Virtual Box so we don’t need to purchase new hard disk . We will use Virtual Hard Disk concept.

To add new hard disk DataNode must be in “Stopped” state then Follow this steps :-

(A) Open Storage in Settings of DataNode -

(B) Click on “Controller: SATA” & after this click on right “+” icon of “Controller: SATA” :-

© Click On “Create” :-

(D) Then click Next-Next and select the size of virtual disk you want to attach.

  • In my case My Hard Disk size is 20 GiB.

(E) To check hard disk is attached or not run “fdisk -l” command :-

Step 2: Create Partition In virtual Disk 0f DataNode :-

Run this command :-

fdisk /dev/sdb

I want to create 5 GiB partition so that DataNode can only use 5GiB for contribution in Hadoop Cluster.

  • Run this command to check partition :-
fdisk -l /dev/sdb

Run this command to load Driver for Partition :-

udevadm settle
  • Whenever New device is added in Computer then we have to load respectively driver so that we can communicate with that device.

Step 3: Format & Mount Partition at DataNode

(A) Run this command to format :-

mkfs.ext4 /dev/sdb1
  • In my case I am using “ext4” format type ,you can choose according to you.

(B) Run this command to mount :-

mount /dev/sdb1 /data1

Step 4: Configure NameNode :-

(A) Make a Directory :-

mkdir /namenode

(B) Format the Directory :-

hadoop namenode -format

(C) “hdfs-site.xml” file configuration :-

(D) “core-site.xml” file configuration :-

(E) Starting NameNode :- start namenode

Step 5: Configure DataNode :-

(A) “hdfs-site.xml” file configuration

(B) “core-site.xml” file configuration :-

(C) Starting DataNode :- start datanode

Step 6: Check Contribution Of DataNode In Distributed File Storage of Hadoop Cluster :-

hadoop dfsadmin -report

You can see DataNode is contributing around 5GiB . Thus we can set limitation of contribution of DataNode in Hadoop Cluster.




Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Dev Day 1: The Journey Starts Now

MASH by Envoya: The New Grafana plugins offering Management and Visualization for Kafka Clusters

Duality 2021 Year-End Review


Amazon’s Alexa meets DevOps (2/2)

Windows 10 Pro Key Generator Reddit

An Intro to Git and GitHub for Beginners | Git for Newbies

Image result for git

A guide on Mining Whive Rewards with your CPU

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store


More from Medium

King Gaskins

Blood, Violence, and Death9.11.01

Wealth Protocol’s Ecosystem