Integrating LVM with Hadoop and providing Elasticity to DataNode Storage
The volume of hadoop Datanode can be increased or decreased using the concept of Elasticity.
Logical Volume Manager (LVM) is a device providing logical volume management for the Linux kernel. Hadoop data nodes shared storage can’t be static so LVM is used to make it dynamic.
LVM Storage Management Structures
LVM functions by layering abstractions on top of physical storage devices. The basic layers that LVM uses, starting with the most primitive, are.
- Physical Volumes
- Volume Groups
- Logical Volumes
LVM can be used to combine physical volumes into volume groups to unify the storage space available on a system. Root volume group into arbitrary logical volumes, which act as flexible partitions.
Prerequisite — Hadoop should be alreday configure i.e. Namenode and Datanode
To integrate LVM with Hadoop and providing Elasticity to DataNode Storage follow the following steps -
Step 1- Add a new hard disk to Datanode
We are adding new hard disk because we don’t have any allocated space to do partitions. After attaching the hard disk check whether the hardisk is properly attached and mounted.
Command : lsblk
or You may use fdisk -l
Step 2- Create Logical Volume
i) Create Physical Group
pvcreate <disk-path>
To display Physical group
pvdisplay <disk-path>
ii) Create Volume Group
vgcreate <vgname> <disk_path>
To display volume group
vgdisplay <vgname> <disk_path>
ii) Create logical volume
lvcreate — size <size> — name <lvname> <vgname>
To display logical volume
lvdisplay — size <size> — name <lvname> <vgname>
Step 3 — Format the logical volume
mkfs.ext4 /dev/<vgname>/<lvname>
Step 4- Mount the Data Node directory with the Logical Volume
mount /dev/<vgname>/<lvname> /<foldername>
Step 5- Restart Data Node
hadoop-daemon.sh stop datanode
hadoop-daemon.sh start datanode
Step 6- Check Contribution Of Data Node In Distributed File Storage of Hadoop Cluster
hadoop dfsadmin -report
Step 7- To increase/decrease the size
lvextend — size +<size> /dev/<vgname>/<lvname>
resize2fs/dev/<vgname>/<lvname>
Step 8- Check the file
Thus, we integrate LVM with Hadoop and provided elasticity to Data node.
Thank you!
