HDFS Commands examples

This article cover simple and useful HDFS commands examples. Apache Hadoop is data processing tool at web scale. Hadoop contains three modules :

HDFS                          ---   Data storage system

MAPREDUCE           ---    Data processing framework

YARN                         ---    Resource management system for Hadoop


We will discuss some commands to learn how to interact with Hadoop distributed File System (HDFS). All hdfs file system commands start with hdfs dfs.  Most of the hadoop distributions  (CDH, HDP) come with standard hdfs user.

1)

Change the current user  root to HDFS user. Mostly hdfs user will be password less user.

2).

Create a file on the local file system using cat command.




3).


Create a directory in HDFS using mkdir command.





4).


Upload local file helloworld to HDFS directory helloworld using put command.

5).

Check file is loaded into hdfs directory helloworld using ls command.

6).

Read hdfs file content using cat command.

7).


Rename HDFS file helloworld to helloworldfile using mv command.

8)

Copy helloworldfile to another hdfs directory using cp command.

9)

Check the size of a file in HDFS using du command.

10).

Check replication factor of a file in HDFS using ls command. Rounded number is the replication factor of the file.

11).

Change the replication factor of a file in HDFS using setrep command.The example below changes replication factor from 3 to 2.