Installing Graphite with Ceres and Megacarbon

Graphite has changed much in the last year, though you wouldn’t know it from reading the official site. Much work has gone in to the new metric writer, Ceres, to improve on the already fantastic Whisper backend which has been the default in the past. Ceres allocates space as metrics are recorded, so your storage needs will scale more directly with use than with Whisper. Additionally, the various carbon daemons (carbon-cache, carbon-relay, etc.) have all been merged into a single daemon called “Megacarbon”. This new daemon supports all the same functionality but streamlines the configuration process. These instructions are for Ubuntu Server 13.04, but you can probably adapt them to your distro of choice if you have sufficient experience.
To start with, you’ll need to get some tools and libraries to help you configure everything. These will need to be installed system-wide:

sudo aptitude install git python-pip python-dev libcairo2 libffi-dev memcached uwsgi nginx uwsgi-plugin-python uwsgi-plugin-carbon
sudo pip install -U pip # Get the latest pip
sudo pip install -U virtualenvwrapper # Get the latest virtualenvwrapper

Now that you’ve got the requirements installed, you’ll need to create a user for graphite:

sudo adduser graphite

You’ll be prompted for details about the user, enter anything you wish. I recommend a 64 or more character password, and that you do not bother to record it. With sudo access to the box, you will not need it. The next step is creating the graphite storage location. I prefer /opt/graphite for this:

sudo mkdir /opt/graphite
sudo chown -R graphite.graphite /opt/graphite
sudo chmod 751 /opt/graphite # More on this later

Now switch to your graphite user and configure virtualenv wrapper:

sudo su - graphite
source /usr/local/bin/
echo "source /usr/local/bin/" >> .bashrc

Once virtualenvwrapper is configured, you can create the new virtualenv and start installing graphite:

mkvirtualenv -a /opt/graphite graphite # This should change your directory to /opt/graphite

We need a directory for our sources:

mkdir src
cd src/

From the src folder, we can start installing graphite components:

export GRAPHITE_ROOT=/opt/graphite
git clone git:// -b megacarbon
cd carbon/
pip install -r requirements.txt
python install
git clone git://
cd ceres/
pip install -r requirements.txt
python install

Next, configure Ceres’ datastore:

ceres-tree-create /opt/graphite/storage/ceres

Now we need to edit config files. Start by copying the example configs:

cd $GRAPHITE_ROOT/conf/carbon-daemons/
cp -r example/ writer
cd writer/

The three files that you should confirm are configured as you desire are daemon.confwriter.conf, and db.conf. It’s a good idea to read the others as well. The parts you must change are:

in daemon.conf:
USER = graphite

in db.conf:
DATABASE = ceres
LOCAL_DATA_DIR = /opt/graphite/storage/ceres/

The other options can remain the same should you so desire. Now we install the web app:

git clone git://
cd graphite-web/

Unfortunately, I wasn’t able to get pycairo to install properly inside a virtualenv, so I used a drop-in replacement library called cairocffi which works flawlessly as far as I can tell. Remove pycairo from the requirements, and install the remainder along with cairocffi:

sed -i '/cairo/d' requirements.txt
pip install -r requirements.txt
pip install cairocffi

Next, we have a small hack to ensure cairocffi gets used instead of pycairo:

echo 'from cairocffi import *' > /home/graphite/.virtualenvs/graphite/lib/python2.7/site-packages/

Finally, install graphite-web:

python install

Now you need an entry point for your application:

cd $GRAPHITE_ROOT/webapp/

Now create in this folder and populate it with the following text:

import os, sys
os.environ['DJANGO_SETTINGS_MODULE'] = 'graphite.settings'

import django.core.handlers.wsgi

app = django.core.handlers.wsgi.WSGIHandler()

This file will be the module imported by uwsgi to run your app. You should take the opportunity now to configure your database. You can create a superuser now as well, who will be the user you log into the application as:

cd $GRAPHITE_ROOT/webapp/graphite/
python syncdb

We’re nearly there. Now we get everything starting on boot and configure nginx/uwsgi:

exit # So we're not running as the graphite user

Create and edit /etc/init/carbon-writer.conf and place the following inside (change eth0 to your adapter name from ifconfig):

description "carbon writer"

start on (net-device-up IFACE=eth0 and local-filesystems)
stop on runlevel [016]

expect daemon

respawn limit 5 10

env GRAPHITE_DIR=/opt/graphite
exec start-stop-daemon --oknodo --pidfile /var/run/ --startas $GRAPHITE_DIR/bin/ --start writer start

Now start the carbon daemon and ensure it’s listening:

sudo start carbon-writer
sudo netstat -plan | grep :2003

You should see a process listening on port 2003. Once that’s up, you can configure the webserver. Create and edit /etc/nginx/sites-available/graphite and enter the following:

server {
  listen 80;
  keepalive_timeout 60;
  charset utf-8;
  location / {
    include uwsgi_params;
    uwsgi_param UWSGI_CHDIR /opt/graphite/webapp/;
    uwsgi_param UWSGI_PYHOME /home/graphite/.virtualenvs/graphite/;
    uwsgi_param UWSGI_MODULE wsgi;
    uwsgi_param UWSGI_CALLABLE app;
  location /content/ {
    alias /opt/graphite/webapp/content/;
    autoindex off;

Naturally you should substitute for your actual domain. Now we ensure nginx can read the content directory (this is why we set /opt/graphite to permission 751 earlier):

sudo chown -R graphite.www-data /opt/graphite/webapp/content
sudo chmod 751 /opt/graphite/webapp /opt/graphite/webapp/content

Now we create a configuration file for uwsgi. Create and edit /etc/uwsgi/apps-available/graphite.ini and enter the following:

plugins = python
gid = graphite
uid = graphite
vhost = true
socket =
master = true
processes = 4
harakiri = 20
limit-as = 256
wsgi-file = /opt/graphite/webapp/
virtualenv = /home/graphite/.virtualenvs/graphite
chdir = /opt/graphite
carbon =

Note the carbon directive, which tells uwsgi to graph some metrics about accesses, memory, etc. This will be our test data to ensure everything works.

By default in Ubuntu, uwsgi will run as the www-data user and cannot change privileges to the graphite user. To work around this, we can modify the default configuration at /usr/share/uwsgi/conf/default.ini and remove the uid and gid lines. This will allow the graphite.ini file we just created to drop privileges to the graphite user.  As an important security consideration here, you should ensure that all of your individual uwsgi configs specify a uid/gid if you do this.  Never run exposed daemons as root!

We can now enable the application and load the configurations:

sudo ln -s /etc/nginx/sites-{available,enabled}/graphite
sudo ln -s /etc/uwsgi/apps-{available,enabled}/graphite.ini
sudo service nginx restart
sudo service uwsgi restart

If everything went well, you should now be able to reach your site at the url you configured. When you expand the Graphite group you should see a uwsgi group available that contains metrics about the running instance of uwsgi (graphite itself).

Update July 1st, 2013:

Currently, the ceres maintenance plugins are not included when installing, so it’s not possible to keep your ceres databases cleaned properly (see this issue).  For now, you need to do the following as the graphite user:

cp -rfp src/carbon/plugins/maintenance plugins/maintenance

Then, edit your crontab (crontab -e) and add this line:

7 1 * * * /opt/graphite/bin/ceres-maintenance --configdir=/opt/graphite/conf/carbon-daemons/writer/ rollup

Update 2: March 17th, 2015

Much has changed since I wrote this blog post. Please see akuzmin’s comments below for things he had to do differently from the above instructions. Thank you, akuzmin!

Writing Python like a Jedi

Jedi is an auto-completion library for Python code that I’ve so far really enjoyed using in vim. It’s able to analyze your code (and anything you import) to guess what you mean when you invoke it (Ctrl+Space by default). Out of the box with jedi-vim, the configuration requires a bit more keypressing than I’m generally happy with, so I used Supertab as suggested. It’s advanced enough to know the types of your variables and the argument names for your function calls. I’ve included instructions below that should get you up and running fast.

  1. If you don’t already use it, configure pathogen.vim:
    1. Get pathogen:
      mkdir -p ~/.vim/autoload ~/.vim/bundle; \
      curl -Sso ~/.vim/autoload/pathogen.vim
    2. Only do this if you don’t have an existing vimrc file:
      echo "syntax on
      filetype plugin indent on" > ~/.vimrc
    3. Add pathogen to your .vimrc file:
      sed -i '1i\
      call pathogen#infect()' ~/.vimrc
  2. Install jedi-vim and jedi:
    cd ~/.vim/bundle/; \
    git clone git://; \
    cd jedi-vim; \
    git submodule update --init
  3. Install Supertab:
    cd ~/.vim/bundle/; \
    git clone git://
  4. Configure Supertab:
    echo 'let g:SuperTabDefaultCompletionType = "context"' >> ~/.vimrc
  5. Optional: Disable auto-complete on . (bothers me, up to you)
    echo 'let g:jedi#popup_on_dot = 0' >> ~/.vimrc

At this point, you can begin to type a member of a library and hit <Tab> to automatically suggest ways to finish (or just type <Tab> after a . to show all available options).

MySQL Query Parser

In my day to day activities, I often need to know who’s being especially bad at MySQL on multi-user systems. Sure, running SHOW PROCESSLIST; a few dozen times might give me an idea, but often you just have to start logging and hope for the best.


I built a very basic MySQL query parser in Python that takes a MySQL 5.0/5.1 log file (any size, it’s rather memory efficient) and returns the total number of SELECT, UPDATE, and INSERT queries, tallies their Queries Per Second, and sorts by selects to give a final output of just who’s been bad and in what way.

I’ll let the help output do the talking:

redkrieg@h4xs3rv3r [~]$ --help
usage: [options] [filename]

[filename] is optional and defaults to stdin.


-h, --help show this help message and exit
-v, --verbose Print info on stderr (default: True)
-q, --quiet Suppress stderr output
-o FILE, --output=FILE
Write output to FILE (default: stdout)
-r REGEX, --regex=REGEX
Tally arbitrary REGEX string (slow)

Supports arbitrary regex searches and takes a best-guess approach to identifying the appropriate user when it’s not obvious. Without the regex option, it can chew through a 1GB log in about 30 seconds on my VPS while staying in a single execution thread.

Download the latest version here