Hadoop cluster set up with Ansible.

🔰 Task Description📄

Configure Hadoop and start cluster services using Ansible Playbook

🔰 STEPS:

step 1. Copy java and Hadoop software

step 2. Install java and Hadoop software

step 3. create directory for namenode and datanode

step 4. Configure hdfs-site.xml and core-site.xml file

step 5. format namenode and start services for both the nodes

step 6. check the report

🔰 Practical:

  1. Setup the controller node with installed and configured ansible

Now, create inventory and write managed node ip , user name , password in it.

To, create ansible config file.

2. To check all the hosts

To check , managed nodes are ping or not.

3. Now, Create a playbook for NameNode

- hosts: namenode
vars_files:
- namenode_vars.yml
tasks:
- copy:
src: "/root/jdk-8u281-linux-x64.rpm"
dest: "/root/jdk-8u281-linux-x64.rpm"
- copy:
src: "/root/hadoop-1.2.1-1.x86_64.rpm"
dest: "/root/hadoop-1.2.1-1.x86_64.rpm"
- command:
cmd: "rpm -ivh jdk-8u281-linux-x64.rpm"
- command:
cmd: "rpm -ivh hadoop-1.2.1-1.x86_64.rpm --force"
- lineinfile:
path: "/etc/hadoop/hdfs-site.xml"
regexp: '</configuration>'
insertafter: '<configuration>'
line: "<property>\n<name>dfs.name.dir</name>\n<value>{{ nn_dir }}</value>\n</property>\n</configuration>" - lineinfile:
path: "/etc/hadoop/core-site.xml"
regexp: '</configuration>'
insertafter: '<configuration>'
line: "<property>\n<name>fs.default.name</name>\n<value>hdfs://{{ ansible_facts['default_ipv4']['address'] }}:9001</value>\n</property>\n</configuration>"
- file:
state: directory
path: "{{ nn_dir }}"
- name: Stopping firewall
shell: "systemctl stop firewalld"
- name: Formatting namenode
shell: "echo Y | hadoop namenode -format"
- name: Starting namenode services
shell: "hadoop-daemon.sh start namenode"

Create variable files for namenode.

nn_dir: "/nn"
nn_ip: "192.168.43.245"

Now, run the playbook .

4. Now, create a playbook for DataNode.

- hosts: datanode
vars_files:
- datanode_vars.yml
tasks:
- copy:
src: "/root/jdk-8u281-linux-x64.rpm"
dest: "/root/jdk-8u281-linux-x64.rpm"
- copy:
src: "/root/hadoop-1.2.1-1.x86_64.rpm"
dest: "/root/hadoop-1.2.1-1.x86_64.rpm"
- command:
cmd: "rpm -ivh jdk-8u281-linux-x64.rpm"
- command:
cmd: "rpm -ivh hadoop-1.2.1-1.x86_64.rpm --force"
- lineinfile:
path: "/etc/hadoop/hdfs-site.xml"
regexp: '</configuration>'
insertafter: '<configuration>'
line: "<property>\n<name>dfs.data.dir</name>\n<value>{{ dn_dir }}</value>\n</property>\n</configuration>"
- lineinfile:
path: "/etc/hadoop/core-site.xml"
regexp: '</configuration>'
insertafter: '<configuration>'
line: "<property>\n<name>fs.default.name</name>\n<value>hdfs://{{ nn_ip }}:9001</value>\n</property>\n</configuration>"
- file:
state: directory
path: "{{ dn_dir }}"
- name: Stopping firewall
shell: "systemctl stop firewalld"
- name: Starting namenode services
shell: "hadoop-daemon.sh start datanode"

Create variable files for datanode.

dn_dir: "/dn"
nn_ip: "192.168.43.245"

Now, run the playbook .

5. Now, to check .

Here , softwares are installed and services is also started in both Namenode and Datanode .

6. Now, to check the report of hadoop setup , use

That’s all for this article. Thanks For Reading😊

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Fall 2020 CS373: Week of 28 Sep — 4 Oct

3 Free Ways to Run Android Apps on a Windows 11 PC

SubDAO Receives Investment from Hypersphere

Symfony 4.2 curated new features

Build variant with google play services

Python Tuples vs Lists — Comparison Between Lists and Tuples

Writing Axios Request Interceptors to Retry Web Scraping Failures

Kafka Internals

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Skabhi

Skabhi

More from Medium

Deploying TiDB on Kubernetes

Generate JWK and PASERK easily by “mkkey”

Configure Hadoop and start cluster services using Ansible Playbook

Configure HAProxy Load Balancer Using Ansible Playbook(12.1)