MySQL Query Parser

In my day to day activities, I often need to know who’s being especially bad at MySQL on multi-user systems. Sure, running SHOW PROCESSLIST; a few dozen times might give me an idea, but often you just have to start logging and hope for the best.

NO MORE I SAY!

I built a very basic MySQL query parser in Python that takes a MySQL 5.0/5.1 log file (any size, it’s rather memory efficient) and returns the total number of SELECT, UPDATE, and INSERT queries, tallies their Queries Per Second, and sorts by selects to give a final output of just who’s been bad and in what way.

I’ll let the help output do the talking:

redkrieg@h4xs3rv3r [~]$ queryparser.py --help
usage: queryparser.py [options] [filename]

[filename] is optional and defaults to stdin.

 

options:
-h, --help show this help message and exit
-v, --verbose Print info on stderr (default: True)
-q, --quiet Suppress stderr output
-o FILE, --output=FILE
Write output to FILE (default: stdout)
-r REGEX, --regex=REGEX
Tally arbitrary REGEX string (slow)

Supports arbitrary regex searches and takes a best-guess approach to identifying the appropriate user when it’s not obvious. Without the regex option, it can chew through a 1GB log in about 30 seconds on my VPS while staying in a single execution thread.

Download the latest version here

Posted in Python, Sadmin | Leave a comment

Adding notifications to finch

So, we use XMPP for our internal chat system at work, but I hate pidgin and empathy’s not much better. Naturally I searched for command line alternatives and the least offensive one I could find was finch, which admittedly uses libpurple on the backend, so it’s really pidgin, but at least it’s in a terminal now. Of course, I lost all my notifications of incoming messages, which isn’t cool, so I cooked up a simple script and call it instead of playing sound through finch. In finch’s “sounds” menu, simply change “Automatic” to “command”. In ubuntu, you’ll need two new packages:

sudo aptitude install libnotify-bin mplayer

And you’ll need this script, and to remember the path to it:

#!/bin/bash

latest_line=`find ~/.purple/logs/jabber/ -mtime -1 -printf "%T@ %Tx %TX %p\n" | sort -n -r | head | cut -d ' ' -f 2- | awk '{print $NF}' | head -1 | xargs tail -1 | sed -e 's#<[^>]*>##g'`
mplayer $1 >/dev/null 2>&1 &
notify-send "`echo $latest_line | cut -d ':' -f 3 | awk -F ')' '{print $2}'`" "`echo $latest_line | cut -d ':' -f 4-`"
Posted in Ubuntu | 1 Comment

Pianobar – An open source Pandora client

I’ve been using pianobar for about two weeks now and have yet to experience any crashes that plagued my usage previously. It is by far the most efficient, stable way to use Pandora. In Ubuntu, you can install it from the software center or via “apt-get install pianobar”. I also recommend installing libnotify-bin if you want to use notifications on song changes (instructions below).

Once installed, you can either run it straight away by opening a terminal and typing “pianobar”, or you can set up some configurations first to enhance your experience. I use a shell script to catch events from the player and display notifications with the song artist and title.

For configuration, edit ~/.config/pianobar/config in your favorite editor. Here is my configuration (note that I pay for Pandora One, if you do not, you can’t specify mp3-hifi as your audio_format):

event_command = /home/myuser/bin/pianobar-notify
user = user@domain.com
password = yourpassword
audio_format = mp3-hifi

The script at /home/myuser/bin/pianobar-notify will be called any time an “event” is sent. See the man page for details on how you can extend this. Below is the pianobar-notify script I use (don’t forget to chmod +x):

#!/bin/bash
# create variables
while read L; do
k="`echo "$L" | cut -d '=' -f 1`"
v="`echo "$L" | cut -d '=' -f 2`"
export "$k=$v"
done < <(grep -e '^\(title\|artist\|album\|stationName\|pRet\|pRetStr\|wRet\|wRetStr\|songDuration\|songPlayed\|rating\|coverArt\)=' /dev/stdin) # don't overwrite $1...

case "$1" in
songstart)
# echo 'naughty.notify({title = "pianobar", text = "Now playing: ' "$title" ' by ' "$artist" '"})' | awesome-client -
# echo "$title -- $artist" > $HOME/.config/pianobar/nowplaying
# # or whatever you like...
notify-send "Pianobar - $stationName" "Now Playing: $artist - $title"
;;

# songfinish)
# # scrobble if 75% of song have been played, but only if the song hasn't
# # been banned
# if [ -n "$songDuration" ] &&
# [ $(echo "scale=4; ($songPlayed/$songDuration*100)>50" | bc) -eq 1 ] &&
# [ "$rating" -ne 2 ]; then
# # scrobbler-helper is part of the Audio::Scrobble package at cpan
# # "pia" is the last.fm client identifier of "pianobar", don't use
# # it for anything else, please
# scrobbler-helper -P pia -V 1.0 "$title" "$artist" "$album" "" "" "" "$((songDuration/1000))" &
# fi
# ;;

*)
if [ "$pRet" -ne 1 ]; then
notify-send "Pianobar - ERROR" "$1 failed: $pRetStr"
elif [ "$wRet" -ne 1 ]; then
notify-send "Pianobar - ERROR" "$1 failed: $wRetStr"
fi
;;
esac

Have fun enjoying your music without a web browser!

Posted in Tutorials, Ubuntu | 1 Comment

Deep Fried Philly Steak

In a stroke of pure genius this weekend, my brother created the Deep Fried Philly Steak. For the benefit of mankind, I shall hereby document the process:

Use a frying pan with a touch of olive oil to sautee some vegetables. We used onion, green pepper, mushrooms, and minced garlic, like these:

They might not look so great raw, but fry these bad boys up...

When the vegetables are sauteed to perfection, add sliced beef. We used steak-ums, but I imagine this would be better with something fresher. Fry ’til the beef is done, then add some cheese blend. It’ll look something like this, but more:

Delicious meat... and fat

Next, roll out some roll-up pizza dough, we used the Pillsbury Classic Pizza Crust for ours. Place six slices of provolone on the rolled-out pizza crust and cut in to six squares. Place a ball of your fried treat on each of the slices of provolone:

This is what you're looking for.

Finally, you can top with some mayo if you truly hate your vascular system:

It's a heart attack in the making.

Pull the four corners up to meet in the center so you have four points sticking out, then grab these and fold them up as well. Make sure there aren’t any gaps through which you can see steak, you should have a ball of stretched dough with no holes. Drop this in to the deep fryer, which you should have pre-heated to 375 degrees Farenheit filled with vegetable oil (or lard I guess, but I wanted to be alive to write this).

You can tell it's done when it floats, but I let it go 'til it's good and brown

When it’s done (golden brown and floating, you may need to turn them to get an even fry) pull them out and place them on some paper towels to soak up some of the liquid death in which they sit. You now have the most amazing treat ever created. Enjoy.

Philly steak, in convenient ball form.

Posted in Fun | 2 Comments

Compiling heimdall on Ubuntu 10.10

Heimdall is an open source replacement for the Samsung Galaxy S flashing utility called Odin. I like this because I can’t stand windows, so if I were to have “bricked” my Vibrant, I’d have been pretty well screwed until I found a friend with a windows machine. Now, there’s an option for us “regular” folks that don’t like to pay the microsoft tax just to use our hardware.

First things first, you’ll need some development packages, so run this command from the terminal:
sudo aptitude install build-essential libusb-1.0-0-dev

Next, you’ll need the hiemdall source from here. You’ll want to get the linux source archive.

Now unzip the source to a folder in your home directory, I use ~/source/. There should now be a folder called “Heimdall-Source”.

Change directories to Heimdall-Source and run the following commands:
./configure
make
sudo make install

You now have heimdall installed. The README file in the Heimdall-Source folder has information on use of the tool, but the important things to know are that you must run it with sudo, you’ll need to untar any .tar or .tar.md5 files you have in the odin bundle (tar xvf filename.tar), and include only the parameters from the source that match files you actually have in your odin tarball (if there’s no Spl.bin in your tarball, don’t include the “–secondary Sbl.bin” part of the heimdall command). If your odin bundle contains all of the files heimdall can use, your command would look like this:
sudo heimdall flash --pit s1_odin_20100512.pit --factoryfs factoryfs.rfs --cache cache.rfs --dbdata dbdata.rfs --boot boot.bin --secondary Sbl.bin --param param.lfs --kernel zImage --modem modem.bin

Posted in Android, Ubuntu | Leave a comment

Minimalist fcgid with custom php.ini on cPanel

I recently changed to using fastcgi on my personal cPanel server (11.28, CentOS) and thought others might benefit from my findings, as I wasn’t able to find any complete configuration tutorials.

I was able to replicate the functionality I enjoyed from SuPHP (per directory php.ini configurations) with a simple wrapper script that’s called using an “Action” parameter in the .htaccess file.

To start, you’ll need to run EasyApache:
/scripts/easyapache
I have disabled “Mod SuPHP” from the easyapache configuration and enabled “Mod FCGID”. This was the only change required to enable FastCGI support using the system php.ini configuration.

I reviewed many forum posts on the topic of local php.ini configuration with fcgid, but none really focused on the per-user aspect and how it can be user definable. I accomplished this with a very simple wrapper script for php-cgi that sets the PHPRC shell variable to the desired path, then a handler and action in .htaccess to call the script for handling php files. This allows for different directories to use different php.ini configurations by creating a different wrapper for each php.ini required.

First, edit the wrapper with your favorite text editor:
nano -w ~/public_html/php5.fcgi
Note, the name is arbitrary, but it needs to be in cgi-bin. Add the following code:
#!/bin/sh
PHPRC=~/public_html/
export PHPRC
exec /usr/local/cpanel/cgi-sys/php5

Then save, and make it executable:
chmod 755 ~/public_html/php5.fcgi

Next, you need to associate it with .php (and any other you’d like) extensions. Add these two lines to ~/public_html/.htaccess (or any .htaccess file)
AddHandler php5-fastcgi .php
Action php5-fastcgi /cgi-bin/php5.fcgi

You can add additional extensions to the AddHandler line like so:
AddHandler php5-fastcgi .php .phtml .html

Update: The above instructions will break subdomains and addon domains if a separate php5.fcgi is not placed in each sub/addon cgi-bin folder. To combat this, I’m using a system accessible php5.fcgi that sets PHPRC to ~/public_html/. To do the same:
nano -w /usr/local/cpanel/cgi-sys/php5.fcgi
Then add this code:
#!/bin/sh
PHPRC=~/public_html/
export PHPRC
exec /usr/local/cpanel/cgi-sys/php5

Set permissions correctly:
chown root:wheel /usr/local/cpanel/cgi-sys/php5.fcgi
chmod 755 /usr/local/cpanel/cgi-sys/php5.fcgi

Finally, change .htaccess to use the new path:
AddHandler php5-fastcgi .php
Action php5-fastcgi /cgi-sys/php5.fcgi

That’s it. It’s generic, can be copied to any user account without modification, and uses system configs for fcgid options, so it can use sane defaults.

I’ve uploaded my cpanel3-skel folder with perms set in case you would like to use it on your server.  Click here.

Update:
Helius reports in the comments that the steps above can break Fantastico on the server. To correct this, you’ll need to build cPanel’s PHP:
/scripts/makecpphp
Thanks for the info Helius!

Posted in cPanel, Work | 4 Comments

ADB over SSH – Fun with port forwards

So last week one of my coworkers and I were discussing a VPN configuration to allow adb shells over 3G. Just tonight I realized that a VPN isn’t even required, SSH can do the same job with relatively little effort. Here’s how I rigged my setup.

Prerequisites:

  • A functional SSH server, accessible from outside your network.
  • Connectbot on your phone with a connection configured for your home computer.
  • A functional ADB over USB install with correct PATH variables set. When you open a new command prompt and type “adb shell” you should see a prompt from your phone.

Configuration of the above items are well documented elsewhere and will be outside the scope of this article.

So, having the prerequisites in order, you start by modifying your Connectbot connection as follows:

  • Long-press your connection in the list and choose “Edit host”.
  • Press “Post-login automation”.
  • In the box type “adb connect localhost:5555[ENTER]” the new line here ([ENTER] key) is necessary to automatically run the command.
  • Press “OK”.
  • Press back to return to the connection list.
  • Long-press your connection in the list and choose “Edit port forwards”.
  • Press Menu, then “Add port forward” and enter the following:
    • Nickname: “ADB”.
    • Type: “Remote”.
    • Source port: “5555″
    • Destination: “localhost:5555″
  • Press “Create port forward”.

Your connection is now configured to tunnel the ADB protocol over port 5555 from your computer to the phone via reverse SSH tunnelling. The next step is to configure ADB on the phone to listen over tcp/ip:

  • On your computer, with the phone connected via USB, type “adb tcpip 5555″.

That’s it! You can now disconnect your USB cable, take your phone on a car ride, etc. If you need to use adb with it, simply use Connectbot to connect to your home computer, after the connection is authenticated (you can use password or public-key) the adb connection will be made automatically, so you simply have to use adb as you normally would!

Remember that over 3G you’re going to have some delay for commands, here’s an example push:


redkrieg@h4xt0p:~$ time adb push Downloads/UniversalAndroot-1.6.2-beta5.apk /sdcard/
11 KB/s (964401 bytes in 83.041s)
redkrieg@h4xt0p:~$ ls -hal Downloads/UniversalAndroot-1.6.2-beta5.apk
-rw-r--r-- 1 redkrieg redkrieg 942K 2010-10-08 15:08 Downloads/UniversalAndroot-1.6.2-beta5.apk
redkrieg@h4xt0p:~$

So yeah, it’s slow as dirt, but I totally pushed a file from my home computer to my phone from across the city.

To undo this and restore normal USB ADB functionality, you must first connect with Connectbot using the instructions above, then from your computer run “adb usb” to return the phone to USB mode. If you’re unable to do this, you’ll need to run the following from a local shell on the phone:


su
setprop service.adb.tcp.port -1
stop adbd
start adbd

That’s it. Have fun!

Update: I’ve just thought things through and realized that this leaves root wide open to remote exploit if someone knows your IP. Probably best to switch back to USB adb once you’re done until I figure out a good way to block non-localhost connections to 5555 in iptables on boot. If you know how to do this, please post a comment, otherwise it’ll be my project for the night.

Update 2:
After some testing, it seems like T-mo blocks inbound connections like this, so iptables shouldn’t be necessary. That being said, the following rules should take care of any lingering doubt:


iptables -A INPUT -p tcp --dport 5555 -s 127.0.0.1 -j ACCEPT
iptables -A INPUT -p tcp --dport 5555 -j DROP

Posted in Tutorials | 1 Comment

Gravity part deux.

So I’ve been playing around in Python lately, primarily because I’ve been teaching it to a kid I’ve been tutoring. I decided to take the opportunity to port my old C# based gravity simulation to see how hard it would be to implement. I chose to use pygame because it provides the screen update engine and drawing routines out of the box with an easy way to tell how many milliseconds have passed since the last frame update (required for accurate physics). I haven’t really made any tweaks to the physics engine since the mono version, but it’s still pretty cool to watch. One thing I did add to this is a 2×2 box with alpha 10 that simulates a pencil running over paper wherever a ball passes. I liked the visualization so much, I left it turned on in the version I hereby bequeath to you, dear internet. Below are two of the generated images, the lighter the color of the line, the faster the ball was moving at the time.

First image from my gravity simulator

First image from my gravity simulator

Second image from my gravity simulator

Second image from my gravity simulator

Oh, I suppose you’d like to download the current version. It’s sloppy, but you’re welcome to it! Some code borrowed from the python/pygame tutorials, but the physics are all mine.

Download it here.

Update: Almost forgot!!! Left click adds an additional ball, right click saves a screenshot and clears the background.

Update 2:

This one was after about 30 minutes with 6 balls

This one was after about 30 minutes with 6 balls

Just another image, click to enlarge.

Update 4: Put a video up on YouTube showing what it looks like while running: http://www.youtube.com/watch?v=OOBBPtsu5MM

Posted in Art & Design, Fun | Leave a comment

Evony – what a wonderful game

So, I’ve been playing a new game called Evony lately.  This game is an excellent, free, online game that’s great for even the most casual player.  The only requirement is an internet connection and flash (which 99% of computers already have).  The gameplay mechanics are time based, meaning that every action has a set amount of real world time that the action will take.  What this means for the casual player is that one can log in, start building something, start some research, send off a few attacks, and know that since it’ll take 45 minutes (for example) to do all those things, they can go about their day knowing that they’re not missing out on something because they aren’t sitting in front of the computer all day long!  Casual gamers will find this aspect of Evony very appealing.

What else is great about Evony?  Well for one, it’s massively multiplayer.  At any given time, there are thousands of players on your server, all attacking eachother’s cities, resource plots, and generally having a great time doing it.  Joining an alliance gives players a large group of friends to talk strategy with, get help from (you can reinforce eachother’s cities for instance), and when waiting for more resources to build up, alliance chat makes for some very fun conversation.  Due to the generally slow pace of the game, the social aspect becomes much more pronounced, lending the game more of an us-against-them feel than a me-against-the-computer one.

So, now that I’ve talked a little about what Evony is and why it’s so fun, let me give you a rundown of some things I learned early on that should help new players get started right:

  1. Use your beginner’s protection – When each player starts, they have a 7 day grace period during which they cannot attack or be attacked.  You should plan to spend a large amount of time playing for your first few days, as build times will be very low at the beginning and you’ll have the chance to build up resources very quickly.  Resource plots are vital to the early game, build sawmills, quarrys, iron mines, and farms up to a pretty good level within the first day.
  2. Keep your Town Hall below level 5 – During beginner’s protection, if you get your Town Hall above level 4, you will no longer be protected.  That protection period is vital!
  3. Get your friends to play Evony – If you can get your friends to start playing at the same time you do, you’ll all learn more in the process.  I know that my friends have been able to answer questions I had sooner than I would have been able to find out on my own.  Plus it’s fun to "talk shop" when you’re out and about.
  4. Join an alliance – Many alliances will not take players until they hit a certain level of prestige, but sometimes just being active daily is enough.  Send mail to some of your neighbors and see if they’re looking for people to join with them!
  5. Buy and sell on the market – When you start getting 250k resources and your build queues are full, you should think about buying and selling on the market.  Any time the market has a difference between the highest bid and the lowest offer, there’s room to make some extra gold.  If you have patience, you can make a couple million gold a day.
  6. Capture valleys early – Low level valleys are easy to take with a few hundred archers and will greatly increase your resource production.  In the first few days, you should have all 4 valleys available to you taken up.  I personally prefer forests because of how much lumber will sell for in the market, but you can always abandon a valley and take another if you need to refocus your resource production.
  7. Keep your mayor’s politics high – The higher your mayor’s politics, the more bonus to resource production you’ll get and the less time it will take to build structures.
  8. Use your free speedup – The speedup button doesn’t use any items if the base build time of the structure is less than 5 minutes.  On your first day, most structures will take less than 5 minutes to build.  Free speedups will greatly increase the rate at which you strengthen your town.

So, now that you have some good tips to start out with, go play evony!  You can find me on Server 50, my player name is RedKrieg.

Have fun!

Posted in Fun | Leave a comment

Wheeeeeeeee Gravity!1!!eleven!

Sooooo…. a few days ago I felt like simulating a bit of gravity. Turns out that was the easy part… Basically I wrote a gravity simulator with accurate(?) collision detection all based on force vectors applied over time. I did all the physics and trig work from the top of my head, and the last time I had a physics or trig class was eight years ago, so I might have forgotten something important. Collisions look right to me right now, but the code driving them relies on some pretty loose interpretations of gravity/time, so any frameskips could cause some rather interesting reflections. Tarball (with source) and Ubuntu deb package (binary only) at their respective links.

I made a half-assed attempt at commenting the source, because I’m a nice guy. You should be able to get an idea of what I was thinking from there.

Update [20081002003930EDT]: Almost forgot, uses Cairo for drawing. XOrg process maxes the cpu at the resolution I’m using for this thing… NO idea why. Any programmers wanna help?

Posted in Fun, Ubuntu | 1 Comment