Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

The volume of hadoop Datanode can be increased or decreased using the concept of Elasticity.

Logical Volume Manager (LVM) is a device providing logical volume management for the Linux kernel. Hadoop data nodes shared storage can’t be static so LVM is used to make it dynamic.

LVM Storage Management Structures

LVM functions by layering abstractions on top of physical storage devices. The basic layers that LVM uses, starting with the most primitive, are.

• Physical Volumes
• Volume Groups
• Logical Volumes

LVM can be used to combine physical volumes into volume groups to unify the storage space available on a system. Root volume group into arbitrary logical volumes, which act as flexible partitions.

Prerequisite — Hadoop should be alreday configure i.e. Namenode and Datanode

To integrate LVM with Hadoop and providing Elasticity to DataNode Storage follow the following steps -

Step 1- Add a new hard disk to Datanode

We are adding new hard disk because we don’t have any allocated space to do partitions. After attaching the hard disk check whether the hardisk is properly attached and mounted.

Command : lsblk

or You may use fdisk -l

Step 2- Create Logical Volume

i) Create Physical Group

pvcreate <disk-path>

To display Physical group

pvdisplay <disk-path>

ii) Create Volume Group

vgcreate <vgname> <disk_path>

To display volume group

vgdisplay <vgname> <disk_path>

ii) Create logical volume

lvcreate — size <size> — name <lvname> <vgname>

To display logical volume

lvdisplay — size <size> — name <lvname> <vgname>

Step 3 — Format the logical volume

mkfs.ext4 /dev/<vgname>/<lvname>

Step 4- Mount the Data Node directory with the Logical Volume

mount /dev/<vgname>/<lvname> /<foldername>

Step 5- Restart Data Node

hadoop-daemon.sh stop datanode
hadoop-daemon.sh start datanode

Step 6- Check Contribution Of Data Node In Distributed File Storage of Hadoop Cluster

hadoop dfsadmin -report

Step 7- To increase/decrease the size

lvextend — size +<size> /dev/<vgname>/<lvname>

resize2fs/dev/<vgname>/<lvname>

Step 8- Check the file

Thus, we integrate LVM with Hadoop and provided elasticity to Data node.

Thank you!