Finding Kafka JAR for building Java producers and consumers

File this one under 'why is this so hard?'.

I've started playing with Kafka and after getting a basic cluster up on 3 nodes, I started writing some code to produce my own events. It took me a good 30 minutes to figure out where the Kafka jar is to actually compile some code!

First step is to use the 'sbt' tool to build the installation. I ended up doing a couple of different targets to finally find the jar files.

./sbt update
./sbt package
./sbt package-all

Once all completed, I had to use 'find' to find the jar file to include in an IntelliJ project to compile and run a producer. The jar was finally found under C:\temp\kafka-0.7.2\core\target\scala_2.8.0\kafka-0.7.2.jar

Adding this jar to my IntelliJ project allowed me to compile and run a simple producer.

Setting up log4j rotating logger with MapR Hadoop

This one vexed me for several years and today I finally figured out how to do it!

We have a daemon that runs a Cascading job on request from a client. Since Cascading is build on top of Hadoop, the command to start the daemon looks like any other hadoop request.

The problem was getting the daemon to do a rotating appender in log4j. Since it is a daemon we want it to run lights out 24/7 so just redirecting stdin and stderr to a log file wasn't going to work.

So we wrote a script that sets the following environment variables before launching the daemon:


export HADOOP_LOG_DIR="/home/hadoop/mapr/hadoop_reporting/logs"
export HADOOP_LOGFILE="daemon.log"
export HADOOP_ROOT_LOGGER="INFO,console,DRFA"

The key change is the 'DRFA' line on the logger. You could ignore the first two and the logs would go to the default Hadoop location ($HADOOP_HOME/logs).

This is really useful since we can run different daemons, each writing to their own log files.

Note that this doesn't change the cluster-side logging. They all still go to where they are expected.

Finding "Number of Under-Replicated Blocks" in Hadoop

This one was bugging me for a long time. Even with the cluster idle, the Name Node summary would tell me there were a number of Under-Replicated Blocks in the system.

Turns out that all the Name Node problems we've been having were leaving 'temporary' files in HDFS and for whatever reason when we restarted the Name Node it wouldn't fix them.

I found them under: /log/hadoop/tmp/mapred/staging/<>/.staging/job_*

After confirming that the users weren't running active jobs, removing these directories via the command line reduced the number of blocks in the report and eventually all were cleared.

FYI our Name Node problems APPEAR to have been resolved in Cloudera CDH3 u3. Name Node has been up for 3 days now. Previously we were lucky if it lasted 48 hours.