Log files in Hadoop eco system

In this article , We will learn how to check log files of hadoop daemons and how to read log files of  applications and jobs.

1)  Locate log directory in Apache Ambari

First We need to know log directory for hadoop daemons.  We can use Apache ambari to find out log directory for a service. default log directory for any service is /var/log/.  Many companies may not use default log directory , so it is better to know log directory using either Ambari or Cloudera Manager. We can even use Unix command to find log directories.

The picture below shows log directory for HDFS service.

Click on HDFS  ----> Configs -------> type log in filter box.

The picture below shows how to locate log directory for Apache Oozie using grep command of Unix.

2)  Types of log files

log directories will have three types of files .

.log

Logs of running daemons will be available here in this .log file.

The picture below shows logs of running active name node . tail -f command is used to see live logs of a daemon.

.out

.out file will have startup messages of a daemon. These messages will be useful to trouble shoot startup failures of a daemon.

The picture below show startup messages of active resource manegr.

.log.[date]

Old log files will have date in their name. by default log rotation is daily, so we will see one log per day.

The picture below shows old ranger log files have date in their name.

3) Command for applications logs

We have seen how to check logs of  hadoop daemons, Now we will learn how to check logs of an application.

We need to have application ID to check logs of an application.

The picture below shows how to get logs of an application id application_1513741463894_0007.

Command used : yarn logs -applicationId application_1513741463894_0007

4) Command for job logs.

We can even check the logs of a hadoop job if we have job id.

The picture below shows how to job logs for job_id job_1513741463894_0001.

Command used : mapred job -logs [job_id]

5) Logs from Resource manager UI.

We can even get application logs from resource manager UI. go to Resource manager UI and click on application ID for which you want to check logs.

The picture below shows logs link in Resource manager UI for application application_1513741463894_0007.

Let me know if you have any questions on how to check log files for any service.