Posts tagged with 'Programming'
Friday, 4 March around 4 o'clock pm
ProcLaunch has learned a bunch of new things lately. I've fixed a few bugs and implemented a few new features, including:
- A
--log-level option, so you can set a level other than DEBUG
- Kill profiles that don't exist
- Instead of killing the process and restarting, proclaunch can send it a signal using the
reload file
- Instead of always sending
SIGTERM, the stop_signal file can contain the name of a signal to send when proclaunch wants to stop a profile
- Pid files are properly cleaned up after processes that don't do it themselves
- You won't get two copies of proclaunch if one is already running as root
Get version 1.1 from github! Thanks a bunch to Matt, who hunted down the bugs and helped me figure out the features.
(Also, I added highlight.js syntax highlighting. Hope you like it!)
Tagged:
Programming
Proclaunch
Read More -
permalink
Saturday, 27 November around 6 o'clock pm
I've been using my Mac mini as a torrent downloader and media server for a few years now using Transmission. Tonight I got fed up with my torrents being in one unsorted mass so I learned the HTTP RPC api and wrote a simple blocking client that organizes them.
I tried out an existing client that I found on GitHub, but it was sort of difficult to use, involving EventMachine and evented techniques. If this was part of a long-running processes I would spend the time to learn how to do this stuff, but this is just a simple script so a blocking technique is just fine.
Here's the client class:
require 'rubygems'
require 'mechanize'
require 'json'
class TransmissionClient
attr_reader :agent,
:header_name,
:header_val,
:password,
:port,
:server,
:username
def initialize(params)
@username = params['username']
@password = params['password']
@server = params['server']
@port = params['port'] || "9091"
@agent = Mechanize.new
@agent.auth(username, password)
@header_name = ""
@header_val = ""
end
def server_uri
return "http://#{server}:#{port}/transmission/rpc"
end
def send(method, params)
resp = ""
begin
resp = agent.post(
server_uri,
JSON.generate({
"method" => method,
"arguments" => params
}),
header_name => header_val
)
resp_obj = JSON.parse(resp.body)
if resp_obj["result"] == "success" then
return resp_obj["arguments"]
else
raise RuntimeError resp.body
end
rescue Mechanize::ResponseCodeError => e
@header_name, @header_val =
e.page.search("code").first.content.split(/: /)
retry
end
end
end
The only real surprise here is that Transmission sends a session ID in an error response (http code 409) and expects that session ID to be sent as a header in every request thereafter. Because of how the ruby version of Mechanize works, it's hard to get the headers from the exception that gets thrown on errors, so instead I extract it directly from the response page. Conveniently the page property on the exception acts like a nokogiri object so I can just search for the single code block on the page and grab the conveniently formatted header.
Here's some code that uses the above class to relocate torrents into folders:
def get_folder_from_name(name)
name_parts = name.split(/\./)
series_parts = []
if name_parts[0].match(/^[0-9]+/) then
series_parts.push name_parts.shift
end
name_parts.each do |p|
break if p.match(/(s?)\d/i)
series_parts.push p
end
if series_parts.length > 0 then
return series_parts.map{|s| s.downcase}
.join("_")
.gsub("aaf-", "")
else
return nil
end
end
server_info_str = IO.read("/Users/Peter/.transmission_server");
server_info = JSON.parse(server_info_str)
client = TransmissionClient.new(server_info)
torrents = client.send(
"torrent-get",
"fields" => %w{id name downloadDir}
);
torrents["torrents"].each do |t|
series_name = get_folder_from_name(t["name"])
next unless series_name
name = t['name']
id = t['id']
download_dir = t['download_dir']
puts "Moving #{name} (#{id}) from #{download_dir} to #{series_name}";
client.send(
"torrent-set-location",
"ids" => t["id"],
"location" => "/Users/Peter/Movies/#{series_name}",
"move" => true
)
end
This instanciates a new client and grabs the id, name, and downloadDir fields for each currently active torrent. Then, it extracts a folder name from torrent's name and tells the client to move the torrent into the proper location. The paths are hardcoded because as I said before, this is a pretty stupid simple script. There's no reason they couldn't be in the config file that holds the server info. Also, the function that extracts a folder name is very specific to the torrents I'm working with.
Tagged:
Programming
Ruby
BitTorrent
Read More -
permalink
Thursday, 23 September around 7 o'clock pm
I kind of started ProcLaunch as a lark. Can I actually do better than the existing user space process managers? It turns out that at least a few people think so. I've gotten a ton of great feedback from thijsterlouw, who actually filed bug reports and helped me work through a bunch of issues. ProcLaunch even has some tests now!
As of today, I'm releasing ProcLaunch v1.0, which you can download from the github downloads page. Interesting changes from the initial version:
Moved to an explicit state machine
In the first version there were a lot of edge cases where proclaunch would have a seemingly random sleep, or some other weird thing. I've removed all of the edge cases by creating an explicit state machine. Profiles have a _status() attribute, which is always one of stopped, starting, running, or stopping. The only sleep() is at the end of the main loop.
The main motivation for this change is because the old version was just plain bad design. Every iteration of the main loop woule create a whole new set of Profile objects, overwriting the old list. Awp, but what happens to profiles that should stop? Let's keep track of their pids and keep trying to kill them over and over until they finally die. But what happens if proclaunch dies before those pids die? Do they just live forever, the eternal zombies of a daemon gone wrong?
The new design eliminates both the repeated kill and the overwriting. Now, profiles are kept in a hash keyed on name and are never replaced after creation. Profiles that get stopped are put in the stopping state, which will check up on the pid every second until it finally dies, then moved to stopped, ready to be restarted.
Improved logging
Log lines have a static format: <Timestamp> <Log Level> <Tag> <Message>. <Tag> is either ProcLaunch or the name of the profile. If a message mentiones a pid, it will always be stated as pid <PID>. This change should make it easier to grep through the logs and automatically parse them for monitoring through nagios or what-have-you.
Please check it out and beat it up. If you notice any issues, don't hesitate to submit an issue or email me, or just leave a comment below.
Tagged:
Programming
Proclaunch
Read More -
permalink
Sunday, 22 August around 2 o'clock pm
Browsing around on hacker news one day, I came across a link to a paper entitled "A micro-manual for Lisp - Not the whole truth" by John McCarthy, the self-styled discoverer of Lisp. One commentor stated that they have been using this paper for awhile as a code kata, implementing it several times, each in a different language, in order to better learn that language. The other day I was pretty bored and decided that maybe doing that too would be a good way to learn something and aleviate said boredom. My first implementation is in perl, mostly because I don't want to have to learn a new language and lisp at the same time. The basic start is after the jump.
Tagged:
Perl
Programming
Read More -
permalink
Sunday, 1 August around 6 o'clock pm
Modern web applications are complicated beasts. They've got database processes, web serving processes, and various tiers of actual application services. The first two generally take care of themselves. PostgreSQL, MySQL, Apache, Nginx, lighttpd, they all have well-understood ways of starting and keeping themselves up and running.
But what do you do if you have a bunch of processes that you need to keep running that aren't well understood? What if they're well-understood to crash once in a while and you don't want to have to babysit them? You need a user space process manager. Zed Shaw seems to have coined this term specifically for the Mongrel2 manual, and it describes pretty accurately what you'd want: some user-space program running above init that can launch your processes and start them again if they stop. Dropping privilages would be nice. Oh, and it'd be cool if it were sysadmin-friendly. Oh, and if it could automatically detect code changes and restart that'd be nifty too.
Tagged:
Programming
Perl
Proclaunch
Read More -
permalink
Wednesday, 2 June around 5 o'clock pm
As some of you might know, Twitter provides a streaming API that pumps all of the tweets for a given search to you as they happen. There are other stream variants, including a sample feed (a small percentage of all tweets), "Gardenhose", which is a stastically sound sample, and "Firehose", which is every single tweet. All of them. Not actually all that useful, since you have to have some pretty beefy hardware and a really nice connection to keep up. The filtered stream is much more interesting if you have a target in mind. Since there was such a hubbub about "Lost" a few weeks ago I figured I would gather relevant tweets and see what there was to see. In this first part I'll cover capturing tweets and doing a little basic analysis, and in the second part I'll go over some deeper analysis, including some pretty graphs!
Tagged:
Programming
Perl
Read More -
permalink
Sunday, 30 May around 6 o'clock pm
In my day job I use a mix of perl and C++, along with awk, sed, and various little languages. In our C++ we use a lot of boost, especially simple things like the date_time libraries and tuple. Tuple is a neat little thing, sort of like std::pair except it lets you have up to 10 elements of arbitrary type instead of just the two. One of the major things that it gives you is a correct operator<, which gives you the ability to use it as a key in std::map. Very handy. One tricky thing, though, is generically iterating over every element in the tuple. What then?
Tagged:
Programming
C++
Read More -
permalink
Thursday, 27 May around 8 o'clock pm
Creating actionable information out of raw data is sometimes pretty simple, requiring only small changes. Of the few feature requests that I've received for Calorific, most (all) of them have been for goals. Always listen to the audience, that's my motto! With the latest version you can set up goals like this:
- goals:
- kcal: 2200
- protein: [ 100, 200 ]
- 2010-05-27 breakfast:
- 1000 kcal
- 25 protein
- 2010-05-27 lunch:
- 850 kcal
- 50 protein
- 2010-05-27 lunch:
- 500 kcal
- 50 protein
This example is super simplified, of course, but you can see how it works. Creating an entry with the special name goals with one component for each nutrient you have a goal for. The value of each component is either a single number, which will be taken as a maximum, or a two element range.
Right now these are displayed by changing the color of the values on aggregate reports (daily and weekly). Red means "outside the range" and green means "inside the range".
$ calorific
2010-05-27 <total> 2350 kcal
125 prot
Colors are done using Term::ANSIColor, which is included in core perl. Adding them was fairly easy, because of a simple function colored, which takes a scalar or an arrayref of scalars and a color argument and returns the scalar wrapped in the correct ANSI codes. Future display options could be displaying how much of each nutrient you have left for the day, maybe in a little progress bar type thing. Feature requests and comments are welcome, as always.
Tagged:
Programming
Read More -
permalink
Sunday, 16 May around 11 o'clock pm
Once in a while at Rentrak we have programming competitions, where anyone who wants to, including sysadmins and DBAs, can submit an entry for whatever the problem is. The previous contest involved writing a poker bot which had to play two-card hold'em, while others have involved problems similar in spirit to the Netflix Prize. This time we chose to build virtual robots that shoot each other with virtual cannons and go virtual boom! We'll be using RealTimeBattle, which is a piece of software designed specifically to facilitate contests of this sort. It's kind of like those other robot-battle systems, except instead of requiring you to write your robot in their own arbitrary, broken, horrible language, this lets you write your bot in any language that can talk on stdin and stdout.
Based on my previous entries the natural choice would be perl, right? I thought about it, actually. Started stubbing something out. Wrote some code to emulate enums and it worked on the first try, which brought to light the fact that I hadn't learned a new language in quite a long time and by not using a new language I was missing a golden opportunity. So, which language? The only real constraint that we, the Happy Fun Robot Times Killing Group, decided on was that it had to be easily installable on Ubuntu, which leaves the field pretty much wide open. Ruby? Already know it in passing. Python? Haven't done much with it for a few years but I don't think it's changed that much. Lisp? Hm. Intriging. Clojure looks interesting, and it's a good chance to figure out multithreading.
The RealTimeBattle system is conceptually pretty simple. Your robot is a little doughnut-shaped thing that can go forward, backward, accelerate, brake, and turn. In addition, it has a big cannon and a radar system, both of which can rotate independent of the bot itself. The radar is the only sensor you can rely on, although in some configurations you'll get coordinates relative to your start position every few game ticks.
When the game starts, the system will start up your bot in a child process and attach to stdin and stdout, so from the bot's point of view it's just talking a simple text protocol. In perl, talking this protocol would be a trivial combination of while(<>){ } and print, but in clojure it seems to be a might bit more complicated:
(loop []
(let [in (read-line)]
(if (not (nil? in))
(do
(println in)
(recur)))))
Just writing that bit took me down about a dozen false starts, but I learned a whole lot about clojure in the process so I'm pretty sure it was worth it.
Ok, so now this little bot can listen, let's make it talk. RealTimeBattle has a command that your bot can send to the server to make it print out something in the message log. We can wrap that in a function like so:
(defn message [m & rest]
(println (str "Print " m rest)))
and call that like this:
(message "Hi there my name is Botty McBotterson!")
The two other basic commands that I've implmented so far are Initialize, which will get sent when the system is ready to find out what name your bot has, and GameOption, which tells you all kinds of information about the environment that the bot lives in. Here's the whole program as it stands:
(def game-option-types [
:robot_max_rotate
:robot_cannon_max_rotate
:robot_radar_max_rotate
:robot_min_acceleration
:robot_max_acceleration
:robot_start_energy
:robot_max_energy
:robot_energy_levels
:shot_speed
:shot_min_energy
:shot_max_energy
:shot_energy_increase_speed
:timeout
:debug_level
:send_robot_coordinates])
(def options (ref {}))
(defn message [m & rest]
(println (str "Print " m rest)))
(defn robot-initialize [[first-round]]
(if first-round
(println "Name kabot")))
(defn robot-set-option
[[option-number value]]
(let [option-key (get
game-option-types
(Integer/parseInt option-number))
option-val (Double/parseDouble value)]
(dosync
(alter options (fn [opts] (assoc opts option-key option-val))))
(message (deref options))))
(defn process-input [m]
(let [tokens (seq (.split m " "))
function-name (first tokens)
args (next tokens)]
(message (str function-name " " args))
(cond
(= function-name "Initialize") (robot-initialize args)
(= function-name "GameOption") (robot-set-option args)
:else (message (str function-name " not implemented")))))
(loop []
(let [in (read-line)]
(if (not (nil? in))
(do
(process-input in)
(recur)))))
This is pretty trivial at the moment. My basic design is to have the main thread deal with all of the I/O and updating a global state object, while another thread deals with analyzing this state and figuring out what to do. I haven't decided on any concrete strategies yet but for the first contest it'll probably be pretty stupid.
A few fun things to note: clojure provides very simple interop with Java classes and methods. For example, (.split m " ") calls the split method on m, which is actually just a Java String. The result of that is a String[], which isn't too useful in clojure so we immediately wrap it in a seq, which is sort of like a lazy cons list. Another example of this really trivial interop is the number parsing done in robot-set-option. I figured this out only after about an hour of thrashing about trying to figure out why passing a string as a vector index wasn't DWIMing like it does in perl. This is another example of why I need to do this project in another language. Perl has rotted my brain.
By the way, if there are things that I'm doing in this code that aren't idomatic clojure, please correct me. I just started learning today, after all. I found a pretty good tutorial which has guided me through basic types and stuff, but shortly I'll be branching beyond that into threading and agents and other fun things that it doesn't cover very well.
Tagged:
Programming
Read More -
permalink
Thursday, 8 April around 7 o'clock pm
I'm a nerd. I write software for a living. I spend a lot of my day either sitting in a chair in front of a computer, or laying on my couch using my laptop. I'm not what you'd call... athletic. I did start lifting weights about six months ago but that's really just led to gaining more weight, not losing it. A few years back I started counting calories and I lost some weight, and then stopped counting calories and gained it all back. Time to change that.
Now, I could use one of the many, many online calorie trackers. They're all ok and they have the advantage of being able to enter data whenever and where ever you are, but most of them have ads and using a web interface is kind of slow and staring at ads sucks. Also, the reports you can generate from them are always a bit limited. What if I want to see a monthly average of how many calories I ate as snacks? Or how many calories I shoved down my gullet from fast food? Or maybe I want to track another nutrient, like grams of protein. Doing all of this through a limited web interface would be tricky, to say the least. There has to be a better way.
I've been using this program called ledger for more than three years now to keep track of my finances. The idea is that you maintain a text file that contains all of your transactions in a really simple format, and then you can run basically arbitrary reports on it. I always have emacs open, so maintaining that file is a snap. I'd like to maintain my calorie history in the same way, using a lightly formatted text file. I actually tried to use ledger for this purpose but the syntax just wasn't right. What I really wanted was a way to build up foods from simpler foods, and have those be built from other, simpler foods, all the way down to calories. Something like this:
1 cup milk = 100 kcal
1 scoop protein powder = 65 kcal
1 protein shake =
1.5 cup milk,
2 scoop protein powder
2010-04-08 breakfast
1 protein shake
I danced around this format for quite awhile, trying to parse it line-wise and trying to parse it with Parse::RecDescent and treetop, and nothing ever really fit. Then, I punted. What's a lightweight, human readable format that already has a parser built? Why, YAML of course! Here's the same thing as a YAML snippet:
- 1 cup milk: 100 kcal
- 1 scoop protein powder: 65 kcal
- 1 protein shake:
- 1.5 cup milk
- 2 scoop protein powder
- 2010-04-08 breakfast:
- 1 protein shake
The basic idea revolves around the concept of a recipe. Essentially, a recipe is a count, a label, and a bunch of components that can also be recipes. "100 kcal" is actually a recipe all by itself. Entries are just recipes that have a date instead of a count. At run-time, we resolve all the labels into recipes and then recursively get the values. Ideally everyhing will resolve down to a handful of base units, like "kcal" or "g protein", but if something doesn't resolve it'll get included right into the output.
So, ok, now I just need a program to analyze this stupid thing and print me some reports. That's where Calorific comes in. It's a little application (<500 lines, actually) that parses that YAML file and prints out either a detail or daily report. I have some big plans for it, including a report that gives the monthly average of daily totals, options to limit the date range you want to report, and 30 day moving averages. Installation instructions are in the readme file, if you'd like to try it out.
Tagged:
Perl
Programming
Ledger
Read More -
permalink