Ubuntu – RedKrieg's Blog

Installing Graphite with Ceres and Megacarbon

Graphite has changed much in the last year, though you wouldn’t know it from reading the official site. Much work has gone in to the new metric writer, Ceres, to improve on the already fantastic Whisper backend which has been the default in the past. Ceres allocates space as metrics are recorded, so your storage needs will scale more directly with use than with Whisper. Additionally, the various carbon daemons (carbon-cache, carbon-relay, etc.) have all been merged into a single daemon called “Megacarbon”. This new daemon supports all the same functionality but streamlines the configuration process. These instructions are for Ubuntu Server 13.04, but you can probably adapt them to your distro of choice if you have sufficient experience.
To start with, you’ll need to get some tools and libraries to help you configure everything. These will need to be installed system-wide:

sudo aptitude install git python-pip python-dev libcairo2 libffi-dev memcached uwsgi nginx uwsgi-plugin-python uwsgi-plugin-carbon
sudo pip install -U pip # Get the latest pip
sudo pip install -U virtualenvwrapper # Get the latest virtualenvwrapper

Now that you’ve got the requirements installed, you’ll need to create a user for graphite:

sudo adduser graphite

You’ll be prompted for details about the user, enter anything you wish. I recommend a 64 or more character password, and that you do not bother to record it. With sudo access to the box, you will not need it. The next step is creating the graphite storage location. I prefer /opt/graphite for this:

sudo mkdir /opt/graphite
sudo chown -R graphite.graphite /opt/graphite
sudo chmod 751 /opt/graphite # More on this later

Now switch to your graphite user and configure virtualenv wrapper:

sudo su - graphite
source /usr/local/bin/virtualenvwrapper.sh
echo "source /usr/local/bin/virtualenvwrapper.sh" >> .bashrc

Once virtualenvwrapper is configured, you can create the new virtualenv and start installing graphite:

mkvirtualenv -a /opt/graphite graphite # This should change your directory to /opt/graphite

We need a directory for our sources:

mkdir src
cd src/

From the src folder, we can start installing graphite components:

export GRAPHITE_ROOT=/opt/graphite
git clone git://github.com/graphite-project/carbon.git -b megacarbon
cd carbon/
pip install -r requirements.txt
python setup.py install
cd $GRAPHITE_ROOT/src/
git clone git://github.com/graphite-project/ceres.git
cd ceres/
pip install -r requirements.txt
python setup.py install

Next, configure Ceres’ datastore:

ceres-tree-create /opt/graphite/storage/ceres

Now we need to edit config files. Start by copying the example configs:

cd $GRAPHITE_ROOT/conf/carbon-daemons/
cp -r example/ writer
cd writer/

The three files that you should confirm are configured as you desire are daemon.conf, writer.conf, and db.conf. It’s a good idea to read the others as well. The parts you must change are:

in daemon.conf:
USER = graphite

in db.conf:
DATABASE = ceres
LOCAL_DATA_DIR = /opt/graphite/storage/ceres/

The other options can remain the same should you so desire. Now we install the web app:

cd $GRAPHITE_ROOT/src/
git clone git://github.com/graphite-project/graphite-web.git
cd graphite-web/

Unfortunately, I wasn’t able to get pycairo to install properly inside a virtualenv, so I used a drop-in replacement library called cairocffi which works flawlessly as far as I can tell. Remove pycairo from the requirements, and install the remainder along with cairocffi:

sed -i '/cairo/d' requirements.txt
pip install -r requirements.txt
pip install cairocffi

Next, we have a small hack to ensure cairocffi gets used instead of pycairo:

echo 'from cairocffi import *' > /home/graphite/.virtualenvs/graphite/lib/python2.7/site-packages/cairo.py

Finally, install graphite-web:

python setup.py install

Now you need an entry point for your application:

cd $GRAPHITE_ROOT/webapp/

Now create wsgi.py in this folder and populate it with the following text:

import os, sys
os.environ['DJANGO_SETTINGS_MODULE'] = 'graphite.settings'

import django.core.handlers.wsgi

app = django.core.handlers.wsgi.WSGIHandler()

This file will be the module imported by uwsgi to run your app. You should take the opportunity now to configure your database. You can create a superuser now as well, who will be the user you log into the application as:

cd $GRAPHITE_ROOT/webapp/graphite/
python manage.py syncdb

We’re nearly there. Now we get everything starting on boot and configure nginx/uwsgi:

exit # So we're not running as the graphite user

Create and edit /etc/init/carbon-writer.conf and place the following inside (change eth0 to your adapter name from ifconfig):

description "carbon writer"

start on (net-device-up IFACE=eth0 and local-filesystems)
stop on runlevel [016]

expect daemon

respawn limit 5 10

env GRAPHITE_DIR=/opt/graphite
exec start-stop-daemon --oknodo --pidfile /var/run/carbon-writer.pid --startas $GRAPHITE_DIR/bin/carbon-daemon.py --start writer start

Now start the carbon daemon and ensure it’s listening:

sudo start carbon-writer
sudo netstat -plan | grep :2003

You should see a process listening on port 2003. Once that’s up, you can configure the webserver. Create and edit /etc/nginx/sites-available/graphite and enter the following:

server {
  listen 80;
  keepalive_timeout 60;
  server_name graphite.example.com;
  charset utf-8;
  location / {
    include uwsgi_params;
    uwsgi_param UWSGI_CHDIR /opt/graphite/webapp/;
    uwsgi_param UWSGI_PYHOME /home/graphite/.virtualenvs/graphite/;
    uwsgi_param UWSGI_MODULE wsgi;
    uwsgi_param UWSGI_CALLABLE app;
    uwsgi_pass 127.0.0.1:3031;
  }
  location /content/ {
    alias /opt/graphite/webapp/content/;
    autoindex off;
  }
}

Naturally you should substitute example.com for your actual domain. Now we ensure nginx can read the content directory (this is why we set /opt/graphite to permission 751 earlier):

sudo chown -R graphite.www-data /opt/graphite/webapp/content
sudo chmod 751 /opt/graphite/webapp /opt/graphite/webapp/content

Now we create a configuration file for uwsgi. Create and edit /etc/uwsgi/apps-available/graphite.ini and enter the following:

[uwsgi]
plugins = python
gid = graphite
uid = graphite
vhost = true
logdate
socket = 127.0.0.1:3031
master = true
processes = 4
harakiri = 20
limit-as = 256
wsgi-file = /opt/graphite/webapp/wsgi.py
virtualenv = /home/graphite/.virtualenvs/graphite
chdir = /opt/graphite
memory-report
no-orphans
carbon = 127.0.0.1:2003

Note the carbon directive, which tells uwsgi to graph some metrics about accesses, memory, etc. This will be our test data to ensure everything works.

By default in Ubuntu, uwsgi will run as the www-data user and cannot change privileges to the graphite user. To work around this, we can modify the default configuration at /usr/share/uwsgi/conf/default.ini and remove the uid and gid lines. This will allow the graphite.ini file we just created to drop privileges to the graphite user. As an important security consideration here, you should ensure that all of your individual uwsgi configs specify a uid/gid if you do this. Never run exposed daemons as root!

We can now enable the application and load the configurations:

sudo ln -s /etc/nginx/sites-{available,enabled}/graphite
sudo ln -s /etc/uwsgi/apps-{available,enabled}/graphite.ini
sudo service nginx restart
sudo service uwsgi restart

If everything went well, you should now be able to reach your site at the url you configured. When you expand the Graphite group you should see a uwsgi group available that contains metrics about the running instance of uwsgi (graphite itself).

Update July 1st, 2013:

Currently, the ceres maintenance plugins are not included when installing, so it’s not possible to keep your ceres databases cleaned properly (see this issue). For now, you need to do the following as the graphite user:

cd $GRAPHITE_ROOT/
cp -rfp src/carbon/plugins/maintenance plugins/maintenance

Then, edit your crontab (crontab -e) and add this line:

7 1 * * * /opt/graphite/bin/ceres-maintenance --configdir=/opt/graphite/conf/carbon-daemons/writer/ rollup

Update 2: March 17th, 2015

Much has changed since I wrote this blog post. Please see akuzmin’s comments below for things he had to do differently from the above instructions. Thank you, akuzmin!

Adding notifications to finch

So, we use XMPP for our internal chat system at work, but I hate pidgin and empathy’s not much better. Naturally I searched for command line alternatives and the least offensive one I could find was finch, which admittedly uses libpurple on the backend, so it’s really pidgin, but at least it’s in a terminal now. Of course, I lost all my notifications of incoming messages, which isn’t cool, so I cooked up a simple script and call it instead of playing sound through finch. In finch’s “sounds” menu, simply change “Automatic” to “command”. In ubuntu, you’ll need two new packages:

sudo aptitude install libnotify-bin mplayer

And you’ll need this script, and to remember the path to it:

#!/bin/bash

latest_line=`find ~/.purple/logs/jabber/ -mtime -1 -printf "%T@ %Tx %TX %p\n" | sort -n -r | head | cut -d ' ' -f 2- | awk '{print $NF}' | head -1 | xargs tail -1 | sed -e 's#<[^>]*>##g'`
mplayer $1 >/dev/null 2>&1 &
notify-send "`echo $latest_line | cut -d ':' -f 3 | awk -F ')' '{print $2}'`" "`echo $latest_line | cut -d ':' -f 4-`"

Pianobar – An open source Pandora client

I’ve been using pianobar for about two weeks now and have yet to experience any crashes that plagued my usage previously. It is by far the most efficient, stable way to use Pandora. In Ubuntu, you can install it from the software center or via “apt-get install pianobar”. I also recommend installing libnotify-bin if you want to use notifications on song changes (instructions below).

Once installed, you can either run it straight away by opening a terminal and typing “pianobar”, or you can set up some configurations first to enhance your experience. I use a shell script to catch events from the player and display notifications with the song artist and title.

For configuration, edit ~/.config/pianobar/config in your favorite editor. Here is my configuration (note that I pay for Pandora One, if you do not, you can’t specify mp3-hifi as your audio_format):
event_command = /home/myuser/bin/pianobar-notify user = user@domain.com password = yourpassword audio_format = mp3-hifi

The script at /home/myuser/bin/pianobar-notify will be called any time an “event” is sent. See the man page for details on how you can extend this. Below is the pianobar-notify script I use (don’t forget to chmod +x):
#!/bin/bash # create variables while read L; do k="`echo "$L" | cut -d '=' -f 1`" v="`echo "$L" | cut -d '=' -f 2`" export "$k=$v" done < <(grep -e '^$title\|artist\|album\|stationName\|pRet\|pRetStr\|wRet\|wRetStr\|songDuration\|songPlayed\|rating\|coverArt$=' /dev/stdin) # don't overwrite $1... case "$1" in songstart) # echo 'naughty.notify({title = "pianobar", text = "Now playing: ' "$title" ' by ' "$artist" '"})' | awesome-client - # echo "$title -- $artist" > $HOME/.config/pianobar/nowplaying # # or whatever you like... notify-send "Pianobar - $stationName" "Now Playing: $artist - $title" ;;


#       songfinish)

#               # scrobble if 75% of song have been played, but only if the song hasn't

#               # been banned

#               if [ -n "$songDuration" ] &&

#                               [ $(echo "scale=4; ($songPlayed/$songDuration*100)>50" | bc) -eq 1 ] &&

#                               [ "$rating" -ne 2 ]; then

#                       # scrobbler-helper is part of the Audio::Scrobble package at cpan

#                       # "pia" is the last.fm client identifier of "pianobar", don't use

#                       # it for anything else, please

#                       scrobbler-helper -P pia -V 1.0 "$title" "$artist" "$album" "" "" "" "$((songDuration/1000))" &

#               fi

#               ;;

*) if [ "$pRet" -ne 1 ]; then notify-send "Pianobar - ERROR" "$1 failed: $pRetStr" elif [ "$wRet" -ne 1 ]; then notify-send "Pianobar - ERROR" "$1 failed: $wRetStr" fi ;; esac

Have fun enjoying your music without a web browser!

Compiling heimdall on Ubuntu 10.10

Heimdall is an open source replacement for the Samsung Galaxy S flashing utility called Odin. I like this because I can’t stand windows, so if I were to have “bricked” my Vibrant, I’d have been pretty well screwed until I found a friend with a windows machine. Now, there’s an option for us “regular” folks that don’t like to pay the microsoft tax just to use our hardware.

First things first, you’ll need some development packages, so run this command from the terminal:
sudo aptitude install build-essential libusb-1.0-0-dev

Next, you’ll need the hiemdall source from here. You’ll want to get the linux source archive.

Now unzip the source to a folder in your home directory, I use ~/source/. There should now be a folder called “Heimdall-Source”.

Change directories to Heimdall-Source and run the following commands:
./configure make sudo make install

You now have heimdall installed. The README file in the Heimdall-Source folder has information on use of the tool, but the important things to know are that you must run it with sudo, you’ll need to untar any .tar or .tar.md5 files you have in the odin bundle (tar xvf filename.tar), and include only the parameters from the source that match files you actually have in your odin tarball (if there’s no Spl.bin in your tarball, don’t include the “–secondary Sbl.bin” part of the heimdall command). If your odin bundle contains all of the files heimdall can use, your command would look like this:
sudo heimdall flash --pit s1_odin_20100512.pit --factoryfs factoryfs.rfs --cache cache.rfs --dbdata dbdata.rfs --boot boot.bin --secondary Sbl.bin --param param.lfs --kernel zImage --modem modem.bin

Wheeeeeeeee Gravity!1!!eleven!

Sooooo…. a few days ago I felt like simulating a bit of gravity. Turns out that was the easy part… Basically I wrote a gravity simulator with accurate(?) collision detection all based on force vectors applied over time. I did all the physics and trig work from the top of my head, and the last time I had a physics or trig class was eight years ago, so I might have forgotten something important. Collisions look right to me right now, but the code driving them relies on some pretty loose interpretations of gravity/time, so any frameskips could cause some rather interesting reflections. Tarball (with source) and Ubuntu deb package (binary only) at their respective links.

I made a half-assed attempt at commenting the source, because I’m a nice guy. You should be able to get an idea of what I was thinking from there.

Update [20081002003930EDT]: Almost forgot, uses Cairo for drawing. XOrg process maxes the cpu at the resolution I’m using for this thing… NO idea why. Any programmers wanna help?

I’m at a loss for words

I know I don’t update here often, but I just thought it would be pertinent at this particular junction. This blog is a combination of my personal life (usually just the bad parts for whatever reason) and my technological exploits, and today will only be slightly outside the usual realm of nonsensical debauchery. Today, I am happy. There’s someone beautiful and hilarious and brilliant and amazing who has chosen to grace me with her attention. I can talk to her about the most obscure scientific discovery and she wants to know more about it and how it affects the world. Did I mention amazing? I think I did…

But alas, this wouldn’t be a very good post here at redkrieg.com were I not to include something at least remotely technical… This weekend I finished preliminary work on my own thumbdrive based distribution of ubuntu. I wrote my own sexy splash screen and modified the themes a bit. I added all the usual programs that I love, set my defaults the way I like them, and even fixed a few bugs in the latest version (upon which this was based). Since it’s thumbdrive based and I have an amazing thumbdrive (OCZ ATV Turbo), it boots and runs faster than any hard disk install I’ve ever used. Unfortunately desktop effects only work on intel video hardware, but at least I have a persistent partition upon which to store any changes I make to the OS. My 4GB drive should be enough for a year’s worth of work if I stay away from video. All of this is thanks to two wonderful tools. The first of which is remastersys, which I use to turn a virtual machine into a full distribution of linux. Secondly, and just as important to this particular venture of mine is pendrivelinux.com‘s tutorial on turning ubuntu 7.10 in to a pen drive bootable (and usable) distro. Now the tutorial as is has some problems, all of which are in the zip file near the end of the tutorial. Many entries in that file don’t work, and the defaults are not set for a persistent install. Luckily for you, fearless user, I have below the contents of my syslinux.cfg for your internet related needs. Also, you can put anything you want in isolinux.txt.

syslinux.cfg:
DEFAULT persistent LABEL persistent menu label ^Start Ubuntu in USB persistent mode (saves changes) kernel vmlinuz append file=/preseed/ubuntu.seed boot=casper persistent initrd=initrd.gz vga=0x0317 quiet splash -- LABEL live menu label ^Start Ubuntu in Live mode kernel vmlinuz append file=/preseed/ubuntu.seed boot=casper initrd=initrd.gz vga-0x317 quiet splash -- LABEL xforcevesa menu label Start Ubuntu in safe ^graphics mode kernel vmlinuz append file=/preseed/ubuntu.seed boot=casper persistent xforcevesa initrd=initrd.gz quiet splash -- LABEL oem menu label ^OEM install (for manufacturers) kernel vmlinuz append file=/preseed/ubuntu.seed boot=casper oem-config/enable=true initrd=initrd.gz quiet splash -- LABEL hd menu label ^Boot from first hard disk localboot 0x80 append - DISPLAY isolinux.txt TIMEOUT 50 PROMPT 1

Have fun. I know I am.

Dell tries Ubuntu, I buy a laptop

So, today I got my first laptop in the mail. It’s a Dell Inspiron E1505 preloaded with Ubuntu Linux. I must say, I’m quite a fan of it. Here’s my whole experience thus far:
Continue reading Dell tries Ubuntu, I buy a laptop

Beryl-Project.org

The Beryl Project is a community run branch of Novell’s Compiz, which I have used extensively. Beryl has eschewed such things as stability for the way more fun WOW factor that makes people really want to try it out. The current development is still really stable for me, both on Intel and NVIDIA graphics hardware, and the performance is top notch. I’m running on Ubuntu with custom packages built nightly, but the packages available from the Beryl Project website work great. I think they’re about due for a release soon, so if you’re looking to try something new and exciting on your Linux desktop, get ready for an experience like no other with Beryl. Under Ubuntu 6.10 I had no difficulty installing at all following the advice from their Wiki. You can also get support for Beryl under Ubuntu in the #ubuntu-xgl channel on irc.freenode.net. Give it a try, and if you like it, consider donating. The entire Linux community will thank you!

Ubuntu tips

I’ve been using Ubuntu linux for a nearly six months now… As of last month, I’ve been using it exclusively while at home. This is a small collection of the things I’ve learned.

Automatix
This one I kinda stayed away from for my first couple weeks… I have a tendency to be stubborn and try to do everything myself. Boy was that a mistake! Automatix has been rapidly improving since the release of Dapper Drake (the latest version of Ubuntu) and it’s definitely the easiest way to get all those not-quite-free programs and other things that most people don’t know exist. Get it here. The instructions are fairly detailed and right on the front page. One thing though… don’t install everything unless you’re sure what it is. Trust me. ;D

Compiz
UPDATE: The methods listed here aren’t the best solution any more. Check out this post for more details!

If you have a modern graphics card and you want your desktop to flow, you need XGL/Compiz. There’s a great tutorial for setup here. Compiz is NOT perfect. Hell, it isn’t even close. It’s alpha software in every sense of the word. You will have video artifacting, you will have phantom windows, you will have issues out the wazoo if you try it now, but you know what? I still had more problems with windows than I do with Compiz, and Compiz tells you it’s alpha software… A little food for thought for you. As a side note, I used Method B: Make Xgl Your Standard Display Server for my XGL and Method C: Toggle for my Compiz. While there isn’t exactly an advantage to the Method B for Xgl (it’s just a bit simpler to use), there is a major advantage to using Method C for Compiz… All you do is create the toggle-compiz.sh file, toss it in your /usr/bin folder, make it executable, and put toggle-compiz.sh in your System->Preferences->Sessions on the Startup tab. Now any time you have some graphical issue that requires you to restart Compiz, just open a terminal and type toggle-compiz.sh twice. Problem solved! Plus, any user of the machine can execute the script and use Compiz!

Improve Performance
I’ve used a few of the tweaks on this page to speed things up a bit (not that it’s really necessary). I’m afraid to try InitNG because it’s apparently not 100% stable yet. Now, I’m all about screwing with my X server, because I know my way around a terminal… Messing with the bootup options? One mistake and the machine won’t boot! I of course could just reinstall, but hey, better safe than sorry. I reboot my machine maybe once every two weeks now? Big deal. The best tweak on this page is picking the right kernel. If you’re running Ubuntu for 32 bit machines, pick k7 for AMD processors and i686 for Intels. If you use the 64 bit version, you should already have the right kernel.

Keyboard Shortcuts
There are lots of keyboard shortcuts, but the ones I use most often are Alt+F2 to open a run dialog, and Ctrl+Alt+Left/Right to change desktops. Of course many of the windows key commands work too, like Alt+Tab. Try things til you figure out what you can do! Doesn’t hurt to play around!

Automatic Login
If you’re the only one with physical access to your machine (or you just don’t care if other people get on), you can set up auto-login so you don’t have to type your password to get to the desktop. This doesn’t exempt you from using a password to perform administrative tasks (like using sudo or gksudo), but it saves a hot second. Go to System->Administration->Login Window and check the Security tab. You’ll find it there.

Make AIM links work with GAIM
This one’s nice. Make sure you didn’t install Gaim 2.0 Beta when you were using Automatix earlier, otherwise this tutorial won’t work for you. If you didn’t, you’ll really enjoy the new functionality!

Setting up your MX1000 mouse
I have one of these and I was none too happy that the extra buttons didn’t work out of the box! Luckily I found this entry that gave a very detailed account of how to get the mouse working perfectly. Even sideways scrolling works!

No sound in Firefox?
I’ve heard of some people not having sound in Firefox. No problem! Check out this fix.

Using multiple sound cards
You might have more than one sound card. Maybe one onboard and one pci or usb. I often switch soundcards and it causes problems left and right. I found that if you delete the .asoundrc file from your home directory, everything seems to resolve itself at the next login. If you have issues, give it a shot.

Windows apps
If you absolutely need to play a game that’s only in windows, give wine a shot. Use the info here to install and you should be fine. Wine doesn’t always work, but you never know! If you have sound issues in wine, try the audio fix above… it worked for me when I had no sound at all (multiple sound cards).

Using locate to find files
One thing I always have trouble with is trying to figure out where a certain config file is. Next time you’re trying to find it… try this:

sudo updatedb

locate filename

This is in no way secure (lets anyone see any filename, you might not want that), but its easy, and if you’re the only one with access to the machine? Meh.

Anyway, that’s all I can think of right now. If I remember any other problems I encountered while migrating, I’ll be sure to update this post or at least link to the new one.

Digg this!