Setting up log4j rotating logger with MapR Hadoop

This one vexed me for several years and today I finally figured out how to do it!

We have a daemon that runs a Cascading job on request from a client. Since Cascading is build on top of Hadoop, the command to start the daemon looks like any other hadoop request.

The problem was getting the daemon to do a rotating appender in log4j. Since it is a daemon we want it to run lights out 24/7 so just redirecting stdin and stderr to a log file wasn't going to work.

So we wrote a script that sets the following environment variables before launching the daemon:


export HADOOP_LOG_DIR="/home/hadoop/mapr/hadoop_reporting/logs"
export HADOOP_LOGFILE="daemon.log"
export HADOOP_ROOT_LOGGER="INFO,console,DRFA"

The key change is the 'DRFA' line on the logger. You could ignore the first two and the logs would go to the default Hadoop location ($HADOOP_HOME/logs).

This is really useful since we can run different daemons, each writing to their own log files.

Note that this doesn't change the cluster-side logging. They all still go to where they are expected.