Exporting Cassandra Metrics to Graphite

Anyone that's ever worked with Cassandra knows that metrics are crucial to being able to do any performance tuning. Out of the box, you have the following options for examining metrics in Cassandra:

• nodetool will give you most of the basic metrics, such as read/write latency, heap usage, etc. It's very easy to use. http://wiki.apache.org/cassandra/NodeTool
• OpsCenter - a webapp built by DataStax.

However, chances are you're already running your own monitoring system, such as Ganglia or Graphite. In that case, you may wish to export all of the metric goodness that Cassandra has to offer into your own monitoring system.

Luckily, this isn't hard. You can query JMX directly, or you can query MX4J web interface that Cassandra exposes on top of JMX. I went with the latter approach, and wrote a simply utility that grabs all the available metrics out of MX4J and pipes them into Graphite.

You can checkout the source code and usage here: https://github.com/matvey14/cassandra-utils

You should be able to easily extend the script and push metrics to Ganglia instead by invoking gmetric, if you'd like.

Managing HBase Region Splits

If you're administering HBase, then chances are you've had to think about to manage splits. There's two ways of doing it:

1. Automated splits. This is the default strategy. Regions will be split once they've grown past a certain size. The size is configurable and defaults to 10Gb. 
2. Manual splits. You manually split regions as you see fit. 

Neither of these alternatives are particularly appealing. There's good reasons for managing splits manually, as explained in http://hbase.apache.org/book/important_configurations.html#disable.splitting The major reason why manual splits are advantageous is that you have control over when they are run, and don't have to worry about them changing unpredictably from underneath you. However, manually managing anything is not really a viable option in the world of Big Data.

Luckily, there's a solution that combines the predictability of managing splits manually, with the advantages of automation so you don't have to worry about it. And it's dead simple. I created a script that does essentially the same thing as an automated-strategy, but can be scheduled to run as a cron job.

You can check out the source here: https://github.com/matvey14/hbase-utils

The java code is just a few lines: https://github.com/matvey14/hbase-utils/blob/master/src/matvey14/hbase/tools/HBaseRegionSplitter.java

Sample Usage: ./run_region_splitter.sh -s 10 -r

This will go through all of your HBase regions, and split any region that is bigger than 10Gb. The "-r" argument tells it to actually do splits. Without "-r", the script defaults to "dry-run" mode, so it'll go through each region and show you what will happen, but won't actually do any splitting.