Bugsplat

A Robust Reporting System for Ledger

2012-01-01T10:01:14+00:00

Note: you can find much more information about ledger on ledger-cli.org, including links to official documentation and other implementations

For the last five years I've kept my personal finances in order using the ledger system, a sophisticated command line program that consumes a lightly formatted text file. It's helped me repay debts and get everything in order, helping me financially absorb an injury last month that would have been extremely detrimental just a few years prior.

The stock ledger program is exclusively command-line oriented. For quick checks and greping over output, this is fine. For some time, though, I've wanted a more graphical, more robust way of looking at my finances. I've also wanted a more familiar query language, since version 2.0's queries were someone limited and version 3.0's query syntax is not very well documented yet. Last year I wrote a simple system that pushed monthly reports out to static HTML files, which got me part of the way there but I really wanted something more flexible. Something where I can just write an arbitrary query and have the output dumped to HTML.

Thus, I present Ledger Web. In Ledger Web, your ledger is kept in a text file, just the same as always, reports are ERB files, and queries are SQL. Ledger Web watches your ledger file and whenever it changes dumps it into a PostgreSQL database table. It's also quite customizable, letting you set hooks both before and after row insertion and before and after ledger file load.

Installation

Ledger Web installation is pretty simple. First make sure you have PostgreSQL version 9.0 or greater installed on your machine. Then, run these commands:

$ gem install ledger_web
$ createdb ledger
$ ledger_web

Then, open your web browser to http://localhost:9090/ where you'll see some simple example reports.

Example Report

Let's walk through a simple pair of reports that shows off most of Ledger Web's features. Yesterday I ran across this blog post which draws a comparison between a typical person's budget and a wooden ship, always springing leaks and at risk of sinking to the bottom. I decided to write a report that shows my expenses both summed by year and broken out into individual lines. First, the Leaky Ship report itself:

<% @query = query({:pivot => "Year"}) do %>
select
    account as "Account",
    xtn_year as "Year",
    coalesce(sum(amount), 0) as "Amount"
from
    accounts_years
    left outer join (
        select
            xtn_year,
            account,
            amount
        from
            ledger
    ) x using (account, xtn_year)
where
    account ~ '(Income|Expenses)'
    and xtn_year <= date_trunc('year', cast(:to as date))
group by
    account,
    xtn_year
order by
    account,
    xtn_year
<% end %>
<div class="page-header">
  <h1>Leaky Ship</h1>
</div>
<%= table(@query, :links => {/\d{4}-\d{2}-\d{2}/ =>
    '/reports/register?account=:0&year=:title'}) %>

It starts off with a database query, defined using a helper named query. It uses a table named ledger, which is where your ledger data will be dumped, as well as a view named accounts_years, which is the cross product of every account by every year. This makes sure that rows show up properly even if there's no data for that particular year. Also, it uses :pivot => "Year", which will pivot the report such that each xtn_year will become it's own column.

The :to param in the where clause is automatically populated with the second date in the range at the top of all reports.

Next, it uses some basic Twitter Bootstrap HTML markup to display a nice title, and then uses the table helper to actually dump the query results to an HTML table. The :links option tells the table helper to link the values in any column who's title matches the regular expression /\d{4}-\d{2}-\d{2}/ to /reports/register?account=:0&year=:title, where :0 will get replaced with the value in column 0 (starting from the left, 0 indexed) and :title will be replaced by the title of the column.

Here's a screenshot of what this report looks like (Note: this uses the Stan example ledger that I generated for my previous reporting system):

The register report that Leaky Ship links to is pretty trivial in comparison. Here's the source:

<% expect ['account', 'year'] %>
<% @query = query do %>
   select
       xtn_date as "Date",
       account as "Account",
       note as "Payee",
       amount as "Amount"
   from
       ledger
   where
       xtn_year = :year
       and account = :account
   order by
       xtn_date
<% end %>
<div class="page-header">
  <h1>Register</h1>
</div>
<%= table @query %>

The only thing new that this does is use the expect helper to ensure that account and year are query params. If they are not, expect throws an exception rather than showing bad data. Here's what this one looks like:

Both of these reports, as well as a few others, can be found in my Ledger Web configuration. My config also shows off some of the more advanced customizations you can do.

The README goes into much more detail on how the helpers work and the various config settings work. Please, install it and let me know what you think!

Program Your Finances: Automated Transactions

2011-12-18T18:32:27+00:00

Note: you can find much more information about ledger on ledger-cli.org, including links to official documentation and other implementations

I've been using Ledger for almost five years now to keep track of my personal finances. For three of those years I've lived with a roommate of one form or another. Part of living with a roommate is splitting up bills. Some people decide to do this by dividing the bills up between roommates. For example, Pete pays the electric and gas bills and Andrew pays the water and the cable. Other roommates decide to nominate one person to have all of the bills in their name and post the amounts due every month for everyone to see. This is what my girlfriend and have been doing and it's been working great. All of the bills are in my name and I give her a summary every month and she hands me a check. Easy peasy.

Of course, being a complete and utter nerd means that I have to make this more complicated than it needs to be in the name of reducing the amount of work I have to do.

Automated Transactions

Ledger has an extremely handy feature named automated transactions. The basic idea is that you provide a template transaction and a pattern to match, and ledger will insert the filled-in template transaction every time the pattern matches. Here's an example:

= /Expenses:Utils:/
    $account                        -0.5
    Assets:Receivable                0.5

This instructs ledger to insert a transaction for 50% of the total transaction amount every time a transaction matches the given regexp (/Expenses:Utils:/). The template variable $account will be replaced with the matched account. So if we have this transaction:

2011/12/18 Electric Company
    Expenses:Utils:Electric          $50
    Assets:Checking

ledger will automatically insert this immediately following:

2011/12/18 Electric Company
    Expenses:Utils:Electric         $-25
    Assets:Receivable                $25

I use an automatic transaction identitcal to this one in my personal ledger file to split utilities with my girlfriend. From there I can run a simple report and copy and paste the results into an email once a month.

Virtual Transactions

I wanted to mention another advanced ledger feature that I use every day. For various reasons I keep most of my money in my interest-paying checking account. I have most of it allocated away into various "funds", which are just fake buckets that only exist for me. It's the same idea as ING subaccounts, but implemented in ledger instead of at the bank.

I've implemented these buckets using ledger's virtual transaction feature. Basically, if you surround an account name in square brackets, ledger treats that portion of the transaction as virtual. Ledger will include this transaction in all reports unless you include the --real flag in your report command. Here's an example:

$ ledger bal checking
               $1000  Assets:Checking

Then, we insert this transaction:

2011/12/01 * Establish Emergency Fund
    [Funds:Emergency]                  $1,000.00
    [Assets:Checking]

and run some more reports

$ ledger bal checking funds
                $500  Assets:Checking
                $500  Funds:Emergency
--------------------
               $1000

$ ledger --real bal checking funds
               $1000  Assets:Checking

By our powers combined...

On their own, these two features are pretty useful. It's when you combine them that the awesome power of ledger starts appearing. As some of you may remember, I has a bit of a medical emergency a few weeks ago and being a citizen of these great United States I have private insurance, so of course I'm going to be paying a not-inconsiderable sum out of pocket. How much? Only time will tell. I can't live like that though, I have to put some kind of structure to it or I'll go crazy. So, I looked up my out of pocket maximum and carved out a portion of my emergency fund into a new medical fund:

2011/12/18 * Establish Medical Fund
    [Funds:Medical]                    $4,000
    [Funds:Emergency]

I also added an automatic transaction that will withdraw from the medical fund whenever I record a medical expense:

= /^Expenses:Medical/
    [Funds:Medical]                      -1.0
    [Assets:Checking]                     1.0

Putting it all together, adding a transaction like this:

2011/12/18 * Corner Drug Store
    Expenses:Medical:OTC               $15.00
    Assets:Checking

Will result in these reports:

$ ledger reg funds:medical
11-Dec-01 Establish Medical Fund    [Funds:Medical]       $4000.00   $4000.00
11-Dec-18 Corner Drug Store         [Funds:Medical]        $-15.00   $3985.00

$ ledger reg checking
11-Nov-01 Checking Deposit          Assets:Checking      $10000.00  $10000.00
11-Dec-01 Establish Emergency Fund  [Assets:Checking]    $-5000.00   $5000.00
11-Dec-18 Corner Drug Store         Assets:Checking        $-15.00   $4985.00
                                    [Assets:Checking]       $15.00   $5000.00

$ ledger --real reg checking
11-Nov-01 Checking Deposit          Assets:Checking      $10000.00  $10000.00
11-Dec-18 Corner Drug Store         Assets:Checking        $-15.00   $9985.00

As you can see, the transaction for Corner Drug Store pulled $15 from Assets:Checking which was then automatically replaced from Funds:Medical. The virtual amount available in checking stays the same but the real amount goes down by $15 without any additional input. These two features combined let me spend directly from a virtual account while keeping track of everything for me.

If you go to the Ledger website you can find the manual which has been recently greatly expanded and enhanced. There you'll see that the expression for an automated transaction can be much more advanced if you want it to be. Check it out.

Yet Another (not very) Static Blog Generator

2011-12-14T18:30:26+00:00

The very first post on this blog was about how I wanted a completely static blog and how it'll be great and wonderful and boy howdy was it ever. Over 500 lines of rather dense perl plus almost 20 separate template files because the kind-of-mustache that I decided to implement can't handle inlined templates for loops so I have to do everything as partials.

Needless to say, it isn't very fun to work on. It mostly does what I want but adding new things is pretty painful, as is changing any of the templates. Yesterday I decided that I would see what a Sinatra port would look like. Why Sinatra? It's fun, that's why. Ruby and Sinatra make writing new webapps easy and fun.

Details

The new version is called bugsplat.rb. It's 200 lines of ruby, which is actually more than I wanted but there a lot of functionality packed in there. Here's the complete feature set:

Entries are written in Markdown and checked into the app repo
Entries have a MIME-style header
Entries can have a --fold-- marker that specifies which content should be on the index page
Supports blog posts and static entries that can optionally be linked from the side navigation
Reads all entries into memory at startup
Uses ERB for templates
Caches rendered pages in memory

The production site is hosted on Heroku and uses a unicorn extremely similar to FivePad's setup without the background worker stuff.

Why not some other blog engine?

A while ago I tried porting to Jekyll but without heavy modification I wouldn't have been able to keep the URLs I've built up over the last year and a half. Also for some reason I couldn't wrap my head around liquid templates.

Wordpress or some other dynamic CMS? I could have done that, sure, but that would introduce other dependencies and I really like the emacs-centric workflow of writing markdown files and generating a site. A web-based CMS would have let me write from anywhere but then I'd have to write in the browser, which isn't my idea of fun.

Results

It's not a static site anymore but I think with the caching I have set up it's almost as fast as one. I could even transparently make it one by pre-rendering everything and stashing it in public using a rake task named assets:precompile that Heroku conveniently runs if it exists.

I don't think I'll do that, though. I like the flexibility that this setup gives me.

Re-using

I wouldn't recommend it. There's nothing that precludes someone else forking bugsplat.rb on github, deleting the entries, rewriting the templates, and running their own site, of course, but it wouldn't be trivial. If you actually want to use it on your own site, email me or leave a comment below and we'll work something out.

Another Tiny Webapp

2011-11-30T11:46:50+00:00

Literally ten minutes after hitting the publish button on my last post I took a little tumble and broke a rather important bone in my back, and now I'm on medical leave from work for awhile.

That doesn't stop me from doing fun things, though, so this morning I cooked up a tiny webapp using Sinatra, DataMapper, and Bootstrap that will help me keep track of when I take painkillers. It's called Painkiller Jane after the comic book character.

There's not much interesting going on here to be honest. Basically it's just one database table and an in-app configuration hash that lays out what pills are available, their dosage, and their cooldown period. I can click the buttons when the cooldown is over, but when it's not they tell me what time I can take the next dose.

The only other feature that I might add is a lockout so that it helps me manage which pill to take when because I'm alternating tylenol and advil.

Concurrency on Heroku Cedar

2011-11-27T18:52:36+00:00

I started a small product a few weeks ago called FivePad, a simple easy way to organize your apartment search. It's basically the big apartment search spreadsheet that you and me and everyone we know has made at least three times, except FivePad is way smarter.

The initial versions of FivePad did everything in the web request cycle, including sending email and pulling down web pages. The other day I was about to add my third in-cycle process when I threw up my arms in disgust. The time had come to integrate resque, a great little redis based job queueing system. Except if I ran it the way Heroku makes things easy my costs would get a little bit out of control for a project that isn't making much money yet.

Backstory

First, a little backstory. Earlier in 2011 Heroku announced their new cedar stack, which is a much more general platform for running webapps than their previous platforms. Cedar lets you describe the processes you want to run using a Procfile. Your processes can use one of a large selection of languages, but FivePad is all ruby. Here's what a Procfile can look like:

web: bundle exec rails server
worker: bundle exec rake resque:work QUEUE=*
scheduler: bundle exec ruby ./config/clock.rb

This creates three different types of processes, web, worker, and scheduler. Heroku intends that you run one of each of these on three different dynos which all charge by the hour, but you get 750 hours free every month.

Options

The official way to do this, of course, is to just spin up multiple dynos. Heroku makes this extremely easy:

$ heroku scale web=2 worker=3

Bam. Done. Two web dynos and three worker dynos all running your code and talking to the same database. Gets to be a bit expensive for hobby/tiny projects, though.

Another option is to run multiple Heroku apps with a shared database. This is decent, in that you get multiple full-powered dynos for free. However, it's kind of a pain to manage. You have to deploy to multiple git repos whenever you do a deploy and you have to make sure all of your environment variables are synced. I've tried it, it's not very much fun.

Yet another option is to auto-scale your workers. The basic idea here is that when a job comes in, the resque client triggers a worker to start up in another dyno, process the job, and then shut down when there are no more jobs left. I tried using this for a long time with Remindlyo and experienced an annoying set of race conditions due to how long rails takes to start. Jobs would get lost and dropped kind of randomly.

Cheating With Style

The system that I've devised for FivePad was inspired by this post by Michael van Rooijen. There, he describes how best to run your rails app on Unicorn, a simple pre-forking rack server. Here's my Procfile:

web: unicorn -p $PORT -c ./config/unicorn.rb

Not much to it. Here's what that config/unicorn.rb file looks like:

worker_processes 3
timeout 30

@resque_pid = nil

before_fork do |server, worker|
  @resque_pid ||= spawn("bundle exec rake " + \
  "resque:work QUEUES=scrape,geocode,distance,mailer")
end

This starts out pretty simply. Three worker processes and a 30 second request timeout. But then there's that before_fork hook. This simply runs a specified rake task if and only if it hasn't been run before, immediately prior to forking off the next web worker. In this case, it runs the resque:work task, which is how resque processes jobs.

This will actually result in six processes in each web dyno:

1 unicorn master
3 unicorn web workers
1 resque worker
1 resque child worker when it actually is processing a job

This may be a bit much if your application is super heavy, but for FivePad it's working pretty well. Things are much faster now that all of the heavy duty stuff is done in the background, and scaling up to another dyno automatically scales the workers as well. One thing to consider in the future is to drop the web workers down to two and add another dyno, but I'm not going to do that until it actually has significant revenue coming in.

Another drawback of this is that if the worker falls over for some reason I'll have to restart the whole dyno, but the chances of that happening are pretty low. Resque forks off a child worker for every job it processes, which insulates the master worker from any problems with jobs.

Anyway, for now this is how FivePad is running. Scaling up is will be simple in the future when it's necessary and I can control costs right now when that's really important.

Introducing FivePad

2011-11-17T17:17:12+00:00

For over two years I've been ruminating on an idea for a webapp that would help people coordinate apartment searches amongst roommates. Finding a place is pretty tough, and finding the right place for you and one or more other people is even harder. Emails, phone calls, spreadsheets, links, bookmarks. It's a mess. So, I built FivePad. More details after the fold.

You can add places by simply pasting in a Craigslist URL or typing in the street address, and FivePad takes care of building a page that shows you where it is in the city and the street view of the address. You can add notes to your places, too. If you're searching with roommates you can add invite them and they'll get emails whenever you add stuff, and you'll get emails when they add stuff.

If the place you're looking at has an email address, you can send a note to that address with one click. When they respond their email will get attached to the right place.

The basic product is there but I have a list of features a mile long that I want to add. Expect more postings here as I add stuff :)

It's free to try, so give it a run and let me know what you think.

What sucks about apartment hunting?

2011-10-19T06:31:23+00:00

I want to brainstorm everything that sucks about searching for an apartment. Why is it always so painful? Does it have to suck so much? I have some ideas but I haven't done this in almost a year so the pain, it is not so fresh.

Here's what I've got so far:

Keeping track of everything. Losing phone numbers, did I call that one back, wait is that a duplicate or another unit in the same building?
The uncertainty. Will this landlord suck? How can I find out? Is the neighborhood ok?
Finding some place close. What's the commute time? Where is the nearest grocery store?
Coordinating with roommates or significant others. Will they like it? I guess I could email them...

In the past I've had a few solutions to this, as I'm sure everyone else has too. The first time I searched for a place I didn't really keep track of anything except in my email inbox. It was kind of a disaster. By the time I picked one place I had already lost the others, but didn't find out until after I called them back. The next time was a little better but still I think my roommate and I mostly communicated via email. The third time I built a tiny Rails site for my girlfriend and I to keep track of stuff, but I never really fleshed it out and we kind of just sat next to eachother on the couch and shared craigslist links over IM.

What I'm looking for here is everything else that sucks about searching for a place and why, and what kind of solutions you've duct taped together to make it work. Websites that make any aspect of it easier would also be super helpful. Everybody who responds gets $2.00 of Remindlyo credit :)

Remindlyo is a Go

2011-10-08T17:24:56+00:00

Eight weeks ago I embarked, almost by accident, on one of the most interesting challenges that I've ever set up for myself. I've created something new, something that I don't think the world has had before, and that makes me feel good. So, if you're reading this, go check out Remindlyo, because it's as done as it's going to get for now.

Remindlyo, at it's core, is about procrastination and forgetfulness. Do you forget to call someone important? Do you "forget" because you can't bring yourself to actually dial the phone? Remindlyo helps out with that, since the friction of hitting a single button on a keypad is so much lower than that of dialing the phone in the first place. You've already answered the phone, might as well keep the ball rolling.

For those of you who care about such things, here's the tech stack:

I'm doing this on my own without much feedback, so if you have any please share it. I would really appreciate anything you have to say.

Join the discussion on Hacker News.

Random Remindlyo Things

2011-09-25T15:49:33+00:00

I think everyone is due for an update on my remindlyo progress, but I don't really have an organized post. Here's a bunch of random thoughts instead.

WePay is awesome

Instead of the tried and true but well-known-to-be-horrible PayPal, I've decided to go with WePay for my payments integration. I made this decision last weekend and today I finally hit the "Submit for Approval" button on remindlyo's application page. The staging server made it trivial to write my integration layer, except that I found a bug when trying to test some things. What makes WePay so awesome is the fact that I was able to email some random person at WePay support and they put their team on the problem right away. Gives me great confidence on them having my back when actual production payment-related things go wrong.

GitHub and Heroku are great too

I've been using GitHub for a number of years now but this is the first project where I've extensively used a private repository. Seamless experience all around.

I also have no complains about Heroku. Being able to get this thing up and running and scaleable with basically no efford has be a lifesaver.

Rails? Eh. It's ok.

I've been building remindlyo in rails at the same time I've been working on another side project (Twitter Fiction Reader, go check it out) in sinatra, and while they're both ruby web development frameworks, they're worlds apart. Rails is heavy, opinionated, all-the-batteries-you-will-ever-need-included framework, while sinatra is svelte and agile and fun. The one saving grace of Rails is that there's an awful lot of people out there struggling with it too, so it's easy to find different ways of going about things.

Launching

I'm pretty sure I'm close to launching this sucker. There are only a few things things holding me back right now. WePay, but I'm assuming the people that approve applications don't work on Sunday so hopefully they'll get back to me tomorrow. The only other thing is the marketing site, which needs some love. I might just go buy a template from Theme Forest that includes the bits I need, because making attractive web pages is definitely not something I'm good at.

So, maybe next week? We'll see.

Quadrotor Update 3

2011-09-11T21:06:12+00:00

Quick update on the quadrotor. I went through all of the motor wiring and found that the motors were hooked up to the wrong terminals! [This diagram][multiwii-diagram] plus watching the orientation of the machine on the configuration software helped me figure out where I had gone wrong. After that, I calibrated the throttle range on all of the speed controllers and then tried flying again.

Except, it still doesn't really work. I played around with it for quite awhile but one of the speed controllers is still basically either on or off. No proportional control at all. Also, I relocated the gyro and accelerometer board to dead center of the body, which helped a little.

And then, when I was holding it and playing with the throttle and trim on the transmitter, one of the motors stalled out somehow. I'm wondering if maybe the battery is low again. Going to throw it on the charger for awhile tomorrow and try again.

Quadrotor Update Part 2

2011-09-09T19:39:35+00:00

The new propellers came today. I ordered a pair of sets of 10x4.5 propellers from Amazon and promptly failed to get them to work with the motors. Turns out they're hard to mount! I ended up making a trip to the hardware store to pick up some O-rings, which are much more stable than the little rubber bands the motors came with. Here's what it looks like:

And here's the mess of wires close up:

Awesome, right? Right! I hooked up the battery, turned on the transmitter, and gave it a shot.

Aaaaaaaaand....

It promptly flipped over.

After a few more or less comical runs of this I read up a little bit, and it turns out sometimes this happens because they battery is low. Put it on the charger for awhile and then tried again with no luck. So then I put the small props back on and held the thing in my hand while reving it up, and it turns out one of the motors is way more powerful than the others. I'm not sure why, but I think maybe one of the speed controllers isn't configured right. Now to figure out how to calibrate them without taking everything apart.

In any case, I'm pretty happy about having it together and making some progress. Maybe by this time next week I'll actually have a flying machine!

Quadrotor Update

2011-09-05T11:34:33+00:00

After too many days of everything remindlyo I decided that today would be the day I finally got off my ass and finished putting together my quadrotor. There were a bunch of minor dramas that made me drop it for much too long, all detailed below the fold.

Drama number one was trying to get my Wii Motion Plus knockoff working. It never did, and in fact I'm pretty sure I completely fried it somehow. The software stopped reading the sensor, and some test software I wrote made it apparent that the sensor, even if it was responding to I2C protocol, was only responding with zeroes.

So, that sucked. I eventually pulled myself out of that deep black hole and ordered a replacements sensor from SparkFun. It was pretty expensive but it includes a three axis gyro and a three axis accelerometer on one tiny little board. Ordered, shipped, received. As I was soldering pins onto it I noticed a little 'VCC 3.3v' mark on it. Uh oh. My Arduino is 5v! Back into the deep black hole!

Literally months later I circled back and ordered this little part, a 5v to 3.3v logic level converter. Oh, and a 5v to 3.3v regulator. Oh actually two of each, in case I blew something up again. Took me awhile to get around to actually putting it together, which brings us to today. Soldered up the level converter, hooked everything up, put the propellers on, aaaaaaaaand....

It's too heavy.

The propellers flew off the motors before it moved even an inch above the floor.

Crap.

Anybody have suggestions for better propellers, or how to calculate what kind of propellers I need? As is, the whole thing is 445 grams incluing the battery.

Bootstrapping a Side Business - First Steps

2011-09-04T08:38:05+00:00

For the past few weeks I've been working on a little product that I'm calling remindlyo, which I'm hoping to turn into a secondary income stream. The basic idea is that you put events about the important people in your life, like birthdays, anniversaries, or what have you, into remindlyo. On the day of the event, remindlyo calls you to remind you and connects you to them, all on the same phone call. You can read more about it on the main remindlyo site. In this post I want to talk more about the why instead of the what.

Since I first figured out what a business does, I've wanted to get in on it. I think I was 12 when I first said to my mom "I want to start a business. I don't want a regular job." Of course, like any responsible mother, she replied "How about you go to college and get a few years experience before starting?" Not exactly satisfactory, but it was sound advice. Now, 15 years later, I'm in a position where I can actually follow through.

I've been aware of Patrick McKenzie for a few years now, but I recently came across his Greatest Hits list and over the course of maybe two days I devoured every single post. They're all great. I had already had an idea for a little project like remindlyo, but Patrick's posts inspired me to expand it into something that I could actually sell.

So then the problem was, what to sell? What can I build that people will buy? I've had a few ideas in the past, but then a few weeks ago my girlfriend kind of dropped the idea for remindlyo in my lap one day. When I told another developer at Emma about it he suggested a whole suite of improvements, which pretty much leads directly to the marketing site I have up and running today.

I can hear you asking "but what if people don't buy it?". That's the biggest fear, but also the most irrelevant, at least for me. I'm doing this first and foremost to learn how a business works from the inside. A very close second is the chance a bunch of technologies that I haven't played with until now, including Ruby on Rails, Twilio, and Heroku, as well as how web marketing and the good kind of SEO works. The opportunity to gain an additional income stream is a somewhat distant third.

Program Your Finances: Vacation Tracking

2011-08-04T16:47:48+00:00

Note: you can find much more information about ledger on ledger-cli.org, including links to official documentation and other implementations

Recently my girlfriend and I visited the wonderful city of Vancouver, Canada. While out of country we tend to use my Schwab Investor Checking account because it carries no fees whatsoever, including currency conversions, and it refunds all ATM fees. Last year when we went to Ireland we just kept all of the receipts and figured it out when we got back, which was excrutiatingly painful. Lost receipts, invisible cash transactions, ugh. It hurts to even think about it. This year, I decided to cobble together a simple system so we could track on the fly. Read on to see how it came together.

This system has the following moving parts:

Dropbox
Ledger
Nebulous Notes, a text editor for iOS that syncs with Dropbox
an always-online machine hooked up to Dropbox and capable of running a python script every once in a while

The workflow is pretty simple. Whenever we spent some money, I recorded it in a very simplified manner at the bottom of a ledger file that lives in Dropbox. Here's a few examples:

2011/07/23 Cash
    Cash  160.00 CAD
    ATM Fees  2.50 CAD
    Checking

2011/07/23 SkyTrain tickets
    Transit  5.00 CAD
    Cash

2011/07/24 Acme Cafe
    Food:Breakfast  29.40 CAD
    Checking

Note that, while simple, these are all valid ledger entries. They just have a different account structure than what my main ledger uses. I used a different file and an abbreviated account structure for a few reasons. First, using shorter account names means I didn't have to type as much on the iPhone screen. Second, this was an experiment, so having a separate file means I didn't have to worry about corrupting my main ledger. Third, loading up my main ledger on the phone slowed Nebulous Notes to a crawl, which wouldn't have been fun dealing with on the move.

Nebulous Notes also has a great macro system, so I was able to program a few interesting templates. "Go to the bottom of the file", Food, and Transit were all one button.

So, now that we have transactions going in and being synced, let's get a little fancy. This was a frugal vacation, so I wanted to see totals by account while we were out and about. Here's that python script:

#!/usr/bin/env python

import sys
import re

amount_by_account = {}

ledger_file = sys.argv[1]
summary_file = sys.argv[2]

with open(ledger_file) as f:
    for line in f.readlines():
        if len(line.strip()) == 0:
            continue

        match = re.match("\s+([\w:]+) \s+([0-9.]+) CAD", line)
        if match is not None:
            account = match.group(1)
            amount = match.group(2)

            prev_amount = amount_by_account.get(account, 0.0)

            amount_by_account[account] = prev_amount + \
                float(amount)


with open(summary_file, "w+") as f:
    total = 0
    for account in sorted(amount_by_account.keys()):
        total += amount_by_account[account]
        f.write("{0:<20}{1:>10}\n".format(account, "%.2f" %
            (amount_by_account[account])))

    f.write("------------------------------\n")
    f.write("{0:<20}{1:>10}\n".format("Total", "%.2f" % total))

Why python and not ledger itself? The machine that I had available to run this thing is a PowerPC Mac mini, which I've never been able to get ledger running properly on. So, python it is! Basically, it looks for lines in ledger_file matching the pattern of a ledger posting, totals up the amounts, and prints them out in alphabetical order to summary_file, which also lives on Dropbox. I had this on a one-minute cron schedule, so whenever I wanted to see our totals (and was in wifi range) I could just open summary_file in Nebulous and sync it up.

I could have used one of any number of iOS expense tracking apps, and that might have been the smarter way to go, but what fun would that be? Also, this system automatically backs itself up whenever Nebulous connects to Dropbox AND it lets me do another fun thing: import (almost) directly into my main ledger.

I say "almost" both because of the simplified account structure and because it's in Canadian dollars and my ledger is exclusively US dollars. Problematic! The easiest way to fix this was by applying the power of perl:

#!/usr/bin/perl

use strict;
use warnings;

my %rates = (
    '2011/07/23' => 1.0531,
    '2011/07/24' => 1.0531,
    '2011/07/25' => 1.0595,
    '2011/07/26' => 1.0596,
    '2011/07/27' => 1.0619,
    '2011/07/28' => 1.0615,
    '2011/07/29' => 1.0615,
);

my $current_date = undef;

while (<>) {
    if (/(\d{4}\/\d{2}\/\d{2})/) {
        $current_date = $1;
    }

    if (/(\s+)([\w\d: ]+)  (\d+(\.\d+)?)/) {
        $_ = sprintf(
            "%sExpenses:%s  \$%.2f\n",
            $1,
            $2,
            ($3 * $rates{$current_date})
        );
    }

    s/    Cash/    Expenses:Cash/;

    s/Checking/Assets:Schwab:Checking/;
    print $_
}

Oh perl. So useful. So ugly. For every line, if it looks like a date, save off the date. If it looks like a transaction line, do the currency conversion (rates are calculated by picking a sample transaction from the bank and backing into it). If it looks like a Cash or Checking account, replace it with the long form of the account name. Finally, print it back out. This got me most of the way there, but I had to tweak some of the amounts due to posting lag at the bank and using a different day's conversion rate. I also piped it through ledger to get the formatting cleaned up:

$ convert-vancouver-ledger.pl vancouver_ledger.txt | \
  ledger -f - print > final_ledger_file.txt

A few quick adjustments and sanity checks later and I was able to copy and paste into my main ledger. I also added some metadata so I could get a report for my girlfriend so she could see what we spent and write me a check for half. Frugal, remember? The final transactions ended up looking like this:

2011/07/23 * Cash
    ; :vancouver:
    Expenses:Cash                            $168.51
    Expenses:ATM Fees                          $2.63
    Assets:Schwab:Checking

2011/07/23 * SkyTrain tickets
    ; :vancouver:
    Expenses:Transit                           $5.27
    Expenses:Cash

2011/07/24 * Acme Cafe
    ; :vancouver:
    Expenses:Food:Breakfast                   $31.15
    Assets:Schwab:Checking

I think it turned out pretty well. I spent less than an hour the day after we got home getting everything cleaned up and copied into my main ledger. When we next go out of the country, I'm sure I'll use this same system. I'll probably put the summarizer script in Dropbox, though, so I can tweak it if necessary.

Have you been in this situation and done something different? Leave a comment!

Program your Finances: Reporting for Fun and Profit

2011-07-09T08:14:55+00:00

Note: you can find much more information about ledger on ledger-cli.org, including links to official documentation and other implementations

Another note: I've written a new version of this that is much more dynamic and flexible named Ledger Web.

Last year I wrote what ended up being the most popular article on this blog ever, Program Your Finances: Command-line Accounting. That post went over how I track and report on my finances using a program called Ledger along with a few helper scripts. Recently I expanded that toolset quite a bit and wanted to show how keeping meticulous track of your finances can give you superpowers. Read on for the gory details.

Stan the Example Man

Talking about personal finances is kind of a tricky thing. If you want to give anything more than a cursory treatment of the subject you have to have some data but the closest source of data to hand is always your own. Some people have decided to talk publicly about their data but I'm not quite ready to to that. Instead, I've written a little python tool to generate a plausible but random history when given a simple json config file. Here's a super simple example:

[
    {
        "payee": "Kettleman Bagels",
        "dow": 3,
        "postings": [
            ["Expenses:Food:Breakfast", [7.20, 7.80]],
            ["Assets:Checking"]
        ]
    }
]

This says, "Every day on Thursday, buy breakfast at Kettleman Bagel Company. It should cost between $7.20 and $7.80." The dow key is the day number, where 0 is Monday and 6 is Sunday. The postings array gives a list of Ledger postings that should be inserted for this entry. The first element is the account name, the second is one of: a single float representing the amount in dollars; empty, meaning that this entry should be the balance of all the other entries; or an array of arguments to pass to Python's random.triangular function. There are a bunch more options that I won't get into here but you can see in the github repo.

Using generate.py and this config, I've created a ledger file for a gentleman who we'll call Stan. Why "Stan"? Because he's the man, that's why. Stan is an unattached twenty-something software developer living in Portland, Oregon. He has a car, a moderately sized student loan, and a pretty decent apartment in a so-so area of town. He's been tracking his expenses for almost four years using Ledger, and he's pretty good at it now. (For the curious, Stan is loosely based on me. Simplified in places, exaggerated in others, cheerfully optimistic in salary.)

Reporting? What's that mean?

Collecting all of this data wouldn't be worth a whole lot if I couldn't analyze it in various ways. Ledger lets me look at things lots of really interesting ways, but sometimes it's a little bit too low level. Too nitty gritty. Too miss-forest-for-the-trees. Sometimes I want to step back and get a bigger view of where my financial life has been, and were I can expect it to lead, and maybe where I should make some changes. When a business wants to do this, they create a series of financial reports. Lots of businesses are compelled to do this by the SEC because they're public corporations, but every well-run business will create these reports regularly to help them keep on track.

Well, I'm kind of a business, right? I do work and receive money as the result of that work. I have short and long term debt, investments, equitiy, assets, etc etc. My sole motivation isn't profit, of course, but in a lot of other respects I try to run my finances as if they were a business. To that end, I've made a series of tools that produce a suite of reports that are fairly similar to what a business would want. From top to bottom we have:

Balance sheet A snapshot of important accounts and a general idea of "net worth" over time.
Net worth chart A monthly overview of the "Total" line from the balance sheet for all available months.
Income Statement A monthly breakdown of income, expenses, and liability payments.
Burn Rate Given Stan spends the "Burn" column on average every month for the trailing 12 months and assuming he'll spend about that same amount going forward, his savings will last him "Months" months. He'll run out of money sometime in February 2012.

But Ledger is a command line program!

Ledger is a command-line program, that's true. I couldn't go directly from my ledger file to pretty html reports with charts and tables, so I invoked two of my favorite chainsaws to hack this out: PostgreSQL and python. PostgreSQL is a wonderfully powerful database that happens to be open source and community driven, and also very easy to use. Python is, well, it wouldn't have been my first choice until pretty recently, but now that I've started using it perl has kind of dropped off my radar. It's pretty great.

Here's the outline of how this thing works: 1. Start maintaining a ledger file 1. Create a PostgreSQL database with the ledger schema

Export the ledger to csv using ledger csv and load it into PostgreSQL using load_ledger.sh
Run some sort-of complicated queries and dump them into HTML tables using run_reports.py
Style the html tables using jquery.datatables and build a chart using jqplot

When I started this I knew I wanted a sql database. I chose PostgreSQL in particular over sqlite both out of familiarity, but also because it handles dates so well. Date is a top-level data type in postgres, instead of having to do weird things with strings like in sqlite.

Why a SQL database instead of just futzing with stuff in python data structures? Because in SQL I can express a rotated dataset pretty easily, whereas in python it would have been a lot of code. See run_reports.py for examples of this. Also, it lets me index the hell out of the tables, build summary tables with weird conditions, and still be able to do neat queries.

Neat Queries You Say?

Honestly, with a lot of work these reports could have been expressed using straight ledger without involving the database at all. It would have been nastier and terser and kind of weird, but I could have done it.

Here's a query that ledger would not have been able to do as far as I know, however:

select
    xtn_month,
    sum(case when pay_period = 1 then amount else 0 end) as pp1,
    sum(case when pay_period = 2 then amount else 0 end) as pp2
from
    aggregated_accounts
where
    account ~ 'Expenses'
    and account !~ 'Taxes'
    and account !~ 'Interest'
group by
    xtn_month
where
    xtn_month >= '2011-01-01'
order by
    xtn_month;


 xtn_month  |   pp1   |   pp2   
------------+---------+---------
 2011-01-01 |  418.79 | 1249.39
 2011-02-01 |  477.18 | 1146.11
 2011-03-01 |  432.92 | 1316.65
 2011-04-01 |  439.95 | 1274.56
 2011-05-01 |  385.60 | 1417.73
 2011-06-01 |  547.77 | 1193.86
 2011-07-01 |  189.75 |       0

Being able to group by completely arbitrary things in ledger has been a pain point for me since I started using it. In this case, I'm grouping by pay_period, a column that has this definition in aggregated_accounts:

CASE
    WHEN (
        xtn_date >= '2010-12-05'
        and extract('day' from xtn_date) between 1 and 14
     ) THEN 1
    WHEN (
        xtn_date < '2010-12-05'
        and (
            extract('day' from xtn_date) between 1 and 6
            or extract('day' from xtn_date) between 22 and 31
        )
    ) THEN 1
    ELSE 2
END as pay_period

The "Burn" calculations are another example. Before I had the data in postgres I had an extremely messy shell script that invoked ledger, date, and dc to calculate it, and if anything broke it all fell down with a weird error.

Drawbacks

The only drawback right now is that the postgres/python setup can't handle differing commodities very well. I track my investment accounts in ledger right along side my transactional accounts and ledger has the ability to go download price quotes for the holdings in those accounts whenever you want and do various calculations on them, but the way I'm doing the CSV export right now doesn't do any of that.

Conclusion

With this setup, I'm able to keep my financial data in a simple, easy to use format and retain the ability to do quick checks on it using ledger. In addition, I can do compilcated queries that would get extremely nasty in straight ledger. It's really the best of both worlds. I've put the tools on GitHub if you want to check them out and maybe install them and try them out.

Whew! Did you stick it out? Questions? Comments? Post'em below.

Quadrotor Motors Are Alive!

2011-04-24T14:48:36+00:00

I found some time today to work on my quadrotor project some more. A few weeks ago I got one motor mounted and spinning, just using the RC receiver and trasmitter. Today, I mounted the motors and set up a little test program on the arduino to make them spin. Check it out:

Test program and more info after the fold.

Among other things, this entailed soldering the speed controllers to the motors, mounting the motors, putting more headers on the arduino, and figuring out the basic wiring. Here's the test program:

#include <Servo.h>

#define ARMING_SPEED 900
#define ZERO_SPEED 1300
#define MAX_SPEED 1850

Servo front;
Servo right;
Servo back;
Servo left;

void write_speed(int in_speed) {
  front.writeMicroseconds(in_speed);
  right.writeMicroseconds(in_speed);
  back.writeMicroseconds(in_speed);
  left.writeMicroseconds(in_speed);
}

void setup() {
  front.attach(10);
  right.attach(11);
  back.attach(12);
  left.attach(13);
  write_speed(ARMING_SPEED);
  delay(10000);
}

int speed = 0;
void loop() {
  for(speed = ZERO_SPEED; speed < MAX_SPEED; speed++) {
    write_speed(speed);
    delay(20);
  }
  for(speed = MAX_SPEED; speed > ZERO_SPEED; speed--) {
    write_speed(speed);
    delay(20);
  }
}

All this does is setup four instances of the built-in Servo object on four different pins. After attaching the servo objects to pins, it sets them all to ARMING_SPEED, which is really just a speed that the speed controllers recognize as the throttle being completely off. Then, it waits for 10 seconds and then starts sweeping from ZERO_SPEED (idle but running) to MAX_SPEED (could be up to 2000 but the propellers have a tendency to fall off at that speed).

One note about these speeds. The way an RC receiver controls a servo is via PWM, "pulse width modification". The receiver sends out a train of pulses, each 2000 microseconds apart, to the servo. A width of 1000 indicates "full left", a width of 2000 indicates "full right", and 1500 "centered". A speed controller uses the same protocol, except it can't reverse direction and the range is a little bigger. 1300 is about idle, 2000 is full power, 900 is "safe".

The next step is to get the frame together the rest of the way and mounting the electronics. Oh, and adding in the Wii Motion Plus and Nunchuck boards to get the six axis IMU running. That's for another day, though.

ProcLaunch v1.2

2011-04-15T12:58:59+00:00

Just a few bug fixes this time:

When you send proclaunch SIGHUP, it will send all of the profiles their respective stop signals and then wait for them to shut down. You can tell proclaunch to stop without waiting by sending SIGHUP again.
You can pass the --log-path command line option to change where proclaunch writes it's log. By default this is $profile_dir/error.log

Get it from github! These changes were generously sponsored by Zipline Games, who are using proclaunch to launch lua mongrel2 handlers as part of their Moai Cloud platform.

ProcLaunch Improvements and v1.1

2011-03-04T16:31:23+00:00

ProcLaunch has learned a bunch of new things lately. I've fixed a few bugs and implemented a few new features, including:

A --log-level option, so you can set a level other than DEBUG
Kill profiles that don't exist
Instead of killing the process and restarting, proclaunch can send it a signal using the reload file
Instead of always sending SIGTERM, the stop_signal file can contain the name of a signal to send when proclaunch wants to stop a profile
Pid files are properly cleaned up after processes that don't do it themselves
You won't get two copies of proclaunch if one is already running as root

Get version 1.1 from github! Thanks a bunch to Matt, who hunted down the bugs and helped me figure out the features.

(Also, I added highlight.js syntax highlighting. Hope you like it!)

I Soldered Something!

2011-02-15T17:57:31+00:00

The Arduino is a cool little development board, actually a series of them, that make it a snap to get up and running with embedded development. I've wanted to get my hands on one for awhile but I haven't really had an application. That is, I didn't until I saw this:

And then I did some research and found this:

This is an r/c quadcopter, a four bladed helicopter that uses an Arduino running the MultiWii software wired up to some knock-off Wii sensors to stabilize itself. The concept is similar to an F-22 or F-117, in that the thing is completely unstable and would probably fall out of the sky without computer control. This is then connected to a four channel r/c receiver, controlled by a normal r/c transmitter.

So.

Cool.

Of course I immediately started scheming to get one of these up and running. The biggest problem was that I had no tools, nor an Arduino, nor any r/c equipment, nor materials to put this thing together. Naturally, I turned to SparkFun to set me up. They're a great online store that has all kinds of useful bits, including most everything electronics-wise that I'll need to get this project off the ground.

The box arrived today!

And here's what was inside it:

(click the picture to see Flickr notes)

Basically I needed everything, so I got a cheap but solid soldering iron, solder, wires, headers, a brass "sponge" for cleaning the iron, and of course a pair of Arduino Pro Mini 16MHz 5v as well as the appropriate programming cable.

After unboxing I installed the Arduino software, the USB driver, and tried getting the simple blink example to install on one of the Arduinos without soldering on some headers. Wouldn't program very reliably, so I broke down and actually heated up the iron and melted some stuff. Here's the result:

Most of the joints are fine, but TX0 didn't get much solder through the hole so I'm going to have to watch it. In any case, it works. I got the blink example working and then wrote up a stupid little S-O-S blinking program and installed it. Pretty lame in the grand scheme of things, but I got it all running, including the soldering, in less than an hour. Extremely gratifying and a great start.

Now to buy sensors, motors, propellers, r/c equipment, and batteries, build the frame, etc etc. I'm thinking strongly about refactoring and rewriting large chunks of MultiWii, but that will come after I get it flying with the stock code.

Using the Transmission RPC Interface to Organize Torrents

2010-11-27T18:53:04+00:00

I've been using my Mac mini as a torrent downloader and media server for a few years now using Transmission. Tonight I got fed up with my torrents being in one unsorted mass so I learned the HTTP RPC api and wrote a simple blocking client that organizes them.

I tried out an existing client that I found on GitHub, but it was sort of difficult to use, involving EventMachine and evented techniques. If this was part of a long-running processes I would spend the time to learn how to do this stuff, but this is just a simple script so a blocking technique is just fine.

Here's the client class:

require 'rubygems'
require 'mechanize'
require 'json'

class TransmissionClient
  attr_reader :agent,
              :header_name,
              :header_val,
              :password,
              :port,
              :server,
              :username

  def initialize(params)
    @username = params['username']
    @password = params['password']
    @server   = params['server']
    @port     = params['port'] || "9091"

    @agent = Mechanize.new

    @agent.auth(username, password)

    @header_name = ""
    @header_val = ""
  end

  def server_uri
    return "http://#{server}:#{port}/transmission/rpc"
  end

  def send(method, params)
    resp = ""

    begin 
      resp = agent.post(
        server_uri,
        JSON.generate({
          "method" => method,
          "arguments" => params
        }),
        header_name => header_val
      )

      resp_obj = JSON.parse(resp.body)

      if resp_obj["result"] == "success" then
        return resp_obj["arguments"]
      else
        raise RuntimeError resp.body
      end

    rescue Mechanize::ResponseCodeError => e
      @header_name, @header_val =
          e.page.search("code").first.content.split(/: /)
      retry
    end
  end
end

The only real surprise here is that Transmission sends a session ID in an error response (http code 409) and expects that session ID to be sent as a header in every request thereafter. Because of how the ruby version of Mechanize works, it's hard to get the headers from the exception that gets thrown on errors, so instead I extract it directly from the response page. Conveniently the page property on the exception acts like a nokogiri object so I can just search for the single code block on the page and grab the conveniently formatted header.

Here's some code that uses the above class to relocate torrents into folders:

def get_folder_from_name(name)
  name_parts = name.split(/\./)
  series_parts = []

  if name_parts[0].match(/^[0-9]+/) then
    series_parts.push name_parts.shift
  end

  name_parts.each do |p|
    break if p.match(/(s?)\d/i)
    series_parts.push p
  end

  if series_parts.length > 0 then
    return series_parts.map{|s| s.downcase}
                       .join("_")
                       .gsub("aaf-", "")
  else
    return nil
  end
end

server_info_str = IO.read("/Users/Peter/.transmission_server");
server_info = JSON.parse(server_info_str)

client = TransmissionClient.new(server_info)

torrents = client.send(
  "torrent-get",
  "fields" => %w{id name downloadDir}
);

torrents["torrents"].each do |t|
  series_name = get_folder_from_name(t["name"])
  next unless series_name

  name = t['name']
  id   = t['id']
  download_dir = t['download_dir']

  puts "Moving #{name} (#{id}) from #{download_dir} to #{series_name}";

  client.send(
    "torrent-set-location", 
    "ids"      => t["id"], 
    "location" => "/Users/Peter/Movies/#{series_name}",
    "move"     => true
  )
end

This instanciates a new client and grabs the id, name, and downloadDir fields for each currently active torrent. Then, it extracts a folder name from torrent's name and tells the client to move the torrent into the proper location. The paths are hardcoded because as I said before, this is a pretty stupid simple script. There's no reason they couldn't be in the config file that holds the server info. Also, the function that extracts a folder name is very specific to the torrents I'm working with.

ProcLaunch v1.0

2010-09-23T19:39:47+00:00

I kind of started ProcLaunch as a lark. Can I actually do better than the existing user space process managers? It turns out that at least a few people think so. I've gotten a ton of great feedback from thijsterlouw, who actually filed bug reports and helped me work through a bunch of issues. ProcLaunch even has some tests now!

As of today, I'm releasing ProcLaunch v1.0, which you can download from the github downloads page. Interesting changes from the initial version:

Moved to an explicit state machine

In the first version there were a lot of edge cases where proclaunch would have a seemingly random sleep, or some other weird thing. I've removed all of the edge cases by creating an explicit state machine. Profiles have a _status() attribute, which is always one of stopped, starting, running, or stopping. The only sleep() is at the end of the main loop.

The main motivation for this change is because the old version was just plain bad design. Every iteration of the main loop woule create a whole new set of Profile objects, overwriting the old list. Awp, but what happens to profiles that should stop? Let's keep track of their pids and keep trying to kill them over and over until they finally die. But what happens if proclaunch dies before those pids die? Do they just live forever, the eternal zombies of a daemon gone wrong?

The new design eliminates both the repeated kill and the overwriting. Now, profiles are kept in a hash keyed on name and are never replaced after creation. Profiles that get stopped are put in the stopping state, which will check up on the pid every second until it finally dies, then moved to stopped, ready to be restarted.
Improved logging

Log lines have a static format: <Timestamp> <Log Level> <Tag> <Message>. <Tag> is either ProcLaunch or the name of the profile. If a message mentiones a pid, it will always be stated as pid <PID>. This change should make it easier to grep through the logs and automatically parse them for monitoring through nagios or what-have-you.

Please check it out and beat it up. If you notice any issues, don't hesitate to submit an issue or email me, or just leave a comment below.

Perl with a Lisp

2010-08-22T14:22:36+00:00

Browsing around on hacker news one day, I came across a link to a paper entitled "A micro-manual for Lisp - Not the whole truth" by John McCarthy, the self-styled discoverer of Lisp. One commentor stated that they have been using this paper for awhile as a code kata, implementing it several times, each in a different language, in order to better learn that language. The other day I was pretty bored and decided that maybe doing that too would be a good way to learn something and aleviate said boredom. My first implementation is in perl, mostly because I don't want to have to learn a new language and lisp at the same time. The basic start is after the jump.

Building a lisp seems to center around two key decisions. First, how do you represent your core data structure? A two-element array? A struct? Something a little more complex? Second, what are your scoping rules. Lexical? Dynamic? Global? After that, everything else is gold plating. Substrate-langauge interop, how you represent scopes, how to get closures right, macros, etc, all can be determined later.

I've chosen to write this first implementation in perl. I know perl pretty well but more importantly I don't know lisp very well at all. I've done a little elisp hacking, but not much. I certainly don't know how all of the pieces fit together quite yet. This first post is really more about getting the fundamental data structure and list-manipulation routines and the reader down. Later posts will elaborate on eval and friends, as well as closures, scoping, and perl interop.

Data Structure

Lisp represents most things fundamentally in terms of what's known as a cons cell. This is some sort of object that has two slots for other objects, be they primitives or other cons cells. Being a good little modern perl programmer, I've chosen to implement this as a small Moose-based class:

package Cell;

use Moose;

use overload
    'bool' => sub { return !shift->is_nil() },
    'fallback' => 1
;

has 'car'    => (is => 'rw');
has 'cdr'    => (is => 'rw');
has 'is_nil' => (is => 'ro', default => 0);

1;

Using Moose, we define an object with two read-write slots named car and cdr. This is due entirely to historical precident: car is the first element in the pair, cdr is the second. is_nil is there to allow us to define a fixed nil value later on. The overload allows us to use a Cell in a boolean context. Anything that doesn't have is_nil set is true;

Fundamental Functions

Now that we've got the data structure done, let's define a few fundamental functions to work with it.

our $NIL = Cell->new(is_nil => 1);
sub nil
{
    return $NIL;
}

our $T = "t";
sub t
{
    return $T;
}

sub equal
{
    my ($a, $b) = @_;
    return t if $a eq $b;
    return nil;
}

Notice how $NIL is just hanging out there. It's the only Cell that will ever have _is_nil set. We return the reference to the singleton from the nil function. t is the opposite. We just return the atom t. equal exploits perl's built-in comparison operator eq to compare two things.

Now, the good stuff. List manipulation:

sub cons
{
    my ($thing, $list) = @_;
    return Cell->new(car => $thing, cdr => $list);
}

sub list
{
    reduce { cons($b, $a) } (nil, reverse @_);
}

sub car
{
    my $thing = shift;
    confess "Argument to car must be a list"
        unless ref($thing) && ref($thing) eq 'Cell';
    return defined($thing->car()) ? $thing->car() : nil;
}

sub cdr
{
    my $thing = shift;
    confess "Argument to cdr must be a list"
        unless ref($thing) && ref($thing) eq 'Cell';
    return defined($thing->cdr()) ? $thing->cdr() : nil;
}

cons creates new Cells, setting their car and cdr as appropriate. The list function is a pure convenience thing to make setting up singly-linked lists easy. car and cdr do a small amount of error checking and call out to the given Cell's car() and cdr() methods.

Functions also defines some functions that will be used later, as well as some things that can walk lists and trees made from cons cells and do something with them. It implements a list_string function which will be imported as the (print) function, once we have symbol tables and function importing defined.

There are a bunch of tests for these functions in 01listmanipulation.t.

Reader

Lisp's parser is referred to as the reader. Generally you interact with it using the (read) function, which pulls off of the input stream and returns the next parsed form as an AST. This reader consists of a hand-rolled recursive descent parser in Read.pm that implements these constraints:

Numbers consist only of numeric charcters and decimal points.
Symbols start with [a-zA-Z] and can contain anything within that range, as well as numbers, the ':' character, underscores, and dashes.
String literals start and end with the '"' character. Escaping is not implemented yet.
Lists start with '(', end with ')', and contain one or more whitespace-delimited things.
Whitespace is skipped.

This most basic of readers is only 113 lines of perl, but it can parse a string of characters that look like lisp and turn it into a tree of cons cells, ready to be evaluated. Tests and examples can be found in 02_read.t.

Well, that's all for now. It's a good start, but doesn't really deal with any of the interesting bits yet. Next up: (eval).

What's your personal code kata? Have you written a lisp before? Have any tips for me?

Managing Your Processes with ProcLaunch.

2010-08-08T12:30:37+00:00

Edit 2010-08-08 17:47: ProcLaunch now has a CPAN-compatible install process. See below for details.

I finally got the chance to work some more on proclaunch, my implementation of a user space process manager, like runit or mongrel or god. I wrote up a big overview of the currently available options [previously][12], but in summary: all of the existing options suck. They're either hard to setup, have memory leaks, have a weird configuration language, or are just plain strange. The only viable option was procer, and even that was just sort of a tech demo put together for the Mongrel2 manual.

That's why I started putting together proclaunch. I need some of the features of runit, namely automatic restart, with none of the wackyness, and I wanted it to be easy to automatically configure. I also wanted it to be standalone so I wouldn't have to install a pre-alpha version of Mongrel2 just to manage my own processes.

What of it?

Grab the latest version off of github, unpack it, and run this in the unpacked directory:

$ perl Build.PL
$ ./Build
$ ./Build install

If everything went smoothly you'll have proclaunch somewhere in your path. Now, fire it up:

$ mkdir -p /path/to/some/state/directory
$ sudo proclaunch \
    --debug \
    --foreground \
    /path/to/some/state/directory \
    example_profiles/

If everything goes according to plan, you'll see a bunch of debug info scroll past showing that it scanned the profiles directory, found one called sleeper, and kicked it off. Then, every five seconds you'll see it rescan. If you look in your process list for sleep you'll see bash happily kicking off a sleep 10 in an infinite loop as the nobody user. Now, run this:

$ sudo kill `cat /path/to/some/state/directory/proclaunch.pid`

You should still see the sleep going on, but proclaunch shouldn't show up anywhere. If you launch proclaunch again, you'll see it startup but never start sleeper, since it's already running. This may seem really mundane, but you can't make runit behave this way without some major hacks. Oh, and to actually make proclaunch kill everything before dying, kill it with -HUP:

$ sudo kill -HUP `cat /path/to/some/state/directory/proclaunch.pid`

Now for the automatic restart. Change something about the profiles directory:

$ touch example_profiles/

In the log you should see that proclaunch saw something changed and rescanned immediately. Now change something about sleeper:

$ touch example_profiles/sleeper

Within a few seconds, proclaunch will notice that something happened and restart sleeper. Specifically, it will send sleeper's pid a SIGTERM, wait up to 7 seconds for it to actually die, and then send it a SIGKILL. Now something a little more drastic:

$ sudo mv example_profiles/sleeper example_profiles/sleeper2

proclaunch will notice that sleeper is gone, tell it to stop, then start sleeper2 since it obviously isn't running. You can use this to setup really simple deploys, especially if you're deploying with Capistrano. Just commit your profiles directory to version control and point proclaunch at that directory in the current symlink, making sure that the pid_file is within the deploy directory somewhere. Within 5 seconds of your deploy, proclaunch will see that the inode on the profiles directory changed.

What is a profile, anyway?

If you look in the sleeper directory, you'll see this set of files:

run
pid_file
user

run is a small script that proclaunch expects to execute and have it return in short order, having backgrounded itself and written it's pid to the path contained in pid_file. This forms the core of both proclaunch and procer. Really simple to setup and automate, since there isn't any complicated config file to manage. The user file is special to proclaunch, and tells it what user to start run as. By default, proclaunch will start run as root, which is generally not what you want. procer can do some fun things that proclaunch can't do yet, like manage dependencies between profiles. If there's any demand I'll work on adding that but I don't currently need it.

A small digression into Mac OS X

Initially I wanted to use Privilege::Drop from CPAN to drop privileges when spawning profiles. It's a really clean pure perl module that has no dependencies other than perl itself. It even does a bunch of sanity checking to ensure that the privileges you dropped to are specifically what you wanted to drop to. However, on OS X with perl 5.10, it seems that you can't drop a large number of auxiliary groups that Privilege::Drop doesn't know about, at least not in the way that it's currently written. That's why the code for dropping privileges is inlined in App::ProcLaunch::Profile. It still checks to make sure that the group you tried to drop to is in the list, but it doesn't assert the list matches exactly what you wanted to do.

Is this something you'd use if it were packaged up nicely on CPAN? Want to use it even before that? Let me know in the comments.

Blog Generator Updates

2010-08-06T23:04:47+00:00

I've made some small changes to the way bugsplat.info is generated. First, I refactored publish.pl quite extensively. Instead of being a huge mess of spaghetti-perl, it's nicely factored out into functions, each one doing as little as possible. It got a little longer, but I think it's worth the tradeoff in readability.

Second, I added self-generated shortlinks. Each post on the site has an internal id, which is actually a monotonically increasing sequence number. The short link for a post is http://bugsplat.info/<id>. For this post, it's http://bugsplat.info/13. These are implemented as mod_rewrite rules in .htaccess which are generated using a template, just like every other piece of content on the site.

Third, I wrote a new convenience script named next-entry.pl, the idea for which I shamelessly stole from technosorcery. Basically, it'll prompt me for a post title using bash's read function, then generate a URL and some date strings, as well as comb through the entries/ directory to find the highest id, then increase it by one. It writes all this to a file and then opens emacsclient right at the correct spot to start typing an entry.

Fourth, I worked on the CSS a little bit. Hopefully it looks a little snazzier than it did before.

Daemons are Our Picky, Temperamental Friends

2010-08-01T18:09:00+00:00

Modern web applications are complicated beasts. They've got database processes, web serving processes, and various tiers of actual application services. The first two generally take care of themselves. PostgreSQL, MySQL, Apache, Nginx, lighttpd, they all have well-understood ways of starting and keeping themselves up and running.

But what do you do if you have a bunch of processes that you need to keep running that aren't well understood? What if they're well-understood to crash once in a while and you don't want to have to babysit them? You need a user space process manager. Zed Shaw seems to have coined this term specifically for the Mongrel2 manual, and it describes pretty accurately what you'd want: some user-space program running above init that can launch your processes and start them again if they stop. Dropping privilages would be nice. Oh, and it'd be cool if it were sysadmin-friendly. Oh, and if it could automatically detect code changes and restart that'd be nifty too.

There are quite a few of these things out there, and as Zed points out all of them suck to various degrees. Here's a list of just a few that I've come across.

runit

We actually use runit at work quite a bit. It's... interesting. Essentially you control it through specially-laid-out directories full of named pipes and control files and whatnot. The learning curve is rather steep, especially since it cannot control things that are already daemons, which flies in the face of everything Unix. It's also bizzarely difficult to get started, since it can't daemonize itself.
God

God is a process manager written in ruby. You configure everything with an internal ruby DSL and it takes care of the rest. It'll even kill things when they start taking up too much memory, which is nice, and it looks pretty extensible as far as adding new conditions. It also has a really nice notifications system, with built-in emailing and twittering and campfiring, if that's your thing. Unfortunately, it also looks kind of complicated. You have to have ruby loaded, you have to write your config in ruby, and it's way of loading configs is sort of weird. Oh, and it has memory leaks.
bluepill

Bluepill was written in reaction to god's shortcomings. It's also written in ruby, it's got a ruby DSL, but some things are slightly different. Mostly it's similar to God but without the memory leak, and without the nice notification support.
monit

The industrial-sized solution, monit seems to compete in the same space as Nagios, except with process management tacked on. Big web interface, mostly for whole-system management. I haven't personally tried it.
supervisord

Written in python, supervisord looks more like what we're looking for. It's specifically written for tracking application-level processes. I haven't personally tried it but I've heard nice things. However, the config system looks pretty intimidating, and it doesn't look to have a nice system for managing dynamic configs.
procer

Procer is what started me on this whole adventure. After struggling with runit for almost an entire week, procer was a breath of fresh air. It is structured in the same way as runit, as a directory full of directories full of files. The most basic config is just a directory containing a run script that daemonizes and writes a pid to the path that the pid_file file contains. Procer can also handle dependencies between services, which is nice if process A just has to be running for process B to even start.

Of all of these, procer seems like the easiest to understand and get going with. However, it's sort of a side project inside of the mongrel2 effort and was written specifically for the manual. It doesn't really handle the code changing underneath it. You have to kill off your processes and let procer restart them for you. Also it depends on a core library from mongrel2, which doesn't really make it suitable for other uses.

That being said, I started rolling my own user space process manager yesterday. It's called proclaunch, and it's heavily inspired by procer. Right now it's mainly just a toy. It can launch and restart processes and maintain pid files, but it has no idea how to drop privilages or restart when something changes. Written in core perl with no external dependencies, it should eventually be suitable at least for my specific use cases, and hopefully it will be for yours too.

Have I missed a process manager? Did I misrepresent one of them? Leave a comment and let me know.

Data Mining "Lost" Tweets Part 1

2010-06-02T17:45:00+00:00

As some of you might know, Twitter provides a streaming API that pumps all of the tweets for a given search to you as they happen. There are other stream variants, including a sample feed (a small percentage of all tweets), "Gardenhose", which is a stastically sound sample, and "Firehose", which is every single tweet. All of them. Not actually all that useful, since you have to have some pretty beefy hardware and a really nice connection to keep up. The filtered stream is much more interesting if you have a target in mind. Since there was such a hubbub about "Lost" a few weeks ago I figured I would gather relevant tweets and see what there was to see. In this first part I'll cover capturing tweets and doing a little basic analysis, and in the second part I'll go over some deeper analysis, including some pretty graphs!

Capturing

Let me preface: I have never watched a single episode of "Lost". When it started I had way too much stuff going on to pay attention to television and since then I've sort of conciously stayed away. I pass no judgements on anyone who is a fan or not, or who is evil or not.

The streaming API is pretty easy to work with. You basically give it a comma separated list of search terms and it will give you any and all tweets that match those terms. For example, if you were to run this command:

$ curl -q http://stream.twitter.com/1/statuses/filter.json\?track=bpcares \
    -uYourTwitterName:YourTwitterPass

you would get a stream of semi-humorous tweets about the oil spill.
I wrote a little perl wrapper around curl which will automatically stop capturing after a given number of hours or until it has captured a given number of megabytes. It will also reconnect when the stream dies for any reason. To capture a workable number of tweets, I launched this script on May 23rd at 4:14pm PDT like this:

$ capture-tweet-stream.pl 24 10000 ~/data/lost-finale-tweets.txt \
    'lost,locke,jack,sawyer,smokemonster,theisland,jacob,shepard'

This means, capture any tweets matching those terms for 24 hours or 10 gb, whichever comes first.

A little analysis

For a while as I was running the capture I was tailing the output file and would pause the output whenever a gem of a tweet scrolled past, just so I could retweet it. Here's my favorite two:

If characters who died on #LOST stayed dead, the finale would be nine minutes long.less than a minute ago via webAndy Borowitz
BorowitzReport

I hope Dexter shows up on Lost and kills them all. #FuckLostless than a minute ago via Twitter for iPhoneEd Battes
EdBattes

I happen to be a fan of Dexter, and would have gladly paid money for a crossover. Anyway.

If you want to play along the data is on my dropbox and the code is all on github. First, let's get an idea of how much raw data we're working with. Twitter sends carriage-return separated JSON blobs. Awk to the rescue!

$ gzcat lost-finale-tweets.txt.gz | awk 'BEGIN{RS="\r"}{n+=1}END{print n}'
779750
$

Almost 780,000 tweets. Tweeps were busy! Ok, so what were they saying? A normal approach would be to run through all of the tweets and count up occurances of each word, but because there's so much output I can't do it on my laptop or I'd run out of memory. Instead, here's a map and two stage reduce process. The map is a fairly small perl script that everyone and their mother can pretty much write from memory, the word count mapreduce example:

This one has a few modifications, though. First, it removes all punctuation except '#' and lowercases everything. Second, it will count each individual word as well as each two and three word phrase in the tweet. We can run it like this:

$ gzcat lost-finale-tweets.txt.gz | ./stem.pl | split -l 1000000 - output/out.txt

The reduce happens in two phases, both using this even smaller perl script that just sums the output from the first one:

Which we run like this:

$ find output -exec ./sum.pl {} \; | ./sum.pl | sort -t $'\t' -k 2,2nr > stems.txt

Sort of like a poor man's Hadoop, no? No, you're right. Not really. But it gets the job done, and that's what counts.

Ok, so now we have our word counts. Here's the top 26 words and phrases that people mentioned in these tweets after removing really common english words:

lost    104181
#lost   53188
finale  25322
watching    11204
de lost 10475
tonight 9588
final   9107
lost finale 9101
series  9000
watch   8105
series finale   7487
the lost    7444
jack    5747
episode 5507
lost series 5062
end 4806
lost series finale  4696
watching lost   4179
the lost finale 3631
final de lost   3519
the end 2981
to watch    2964
watching the    2920
spoiler 2804
love    2768
the finale  2765
#lost finale    2579

In amongst all the tiny junk words, we have some really nice indicators that we can use in the next phase to filter to just the tweets that are actually talking about lost the tv show vs their lost kitten named Mittens. Interestingly, the phrase "you all everybody" only showed up 67 times. Sad.

Iterating Elements in boost::tuple, template style

2010-05-30T18:15:00+00:00

In my day job I use a mix of perl and C++, along with awk, sed, and various little languages. In our C++ we use a lot of boost, especially simple things like the date_time libraries and tuple. Tuple is a neat little thing, sort of like std::pair except it lets you have up to 10 elements of arbitrary type instead of just the two. One of the major things that it gives you is a correct operator<, which gives you the ability to use it as a key in std::map. Very handy. One tricky thing, though, is generically iterating over every element in the tuple. What then?

It's easy to get at individual elements when you know how many there are and what their types are:

typedef tuple<int, string, bool> delicious_tuple;
delicious_tuple foo(1, "hi", false);

// get<N>(tuple_type) gives you a reference to the Nth element
cout << get<0>(foo) << endl
     << get<1>(foo) << endl
     << get<2>(foo) << endl;

But what if you don't know those things? A really common situation where this comes up is serialization, where you have a diverse set of tuples and you don't want to write a whole bunch of glue code. tuple overrides operator<< and operator>> for ostreams and istreams, which by default read and write strings:

delicious_tuple foo(2, "there", true);
cout << foo << endl; // prints "(2 another one true)"

Sometimes that just doesn't cut it, though. If you want to serialize to JSON or XML or something, you have to be able to generically get at each element. You could write a macro using the boost preprocessor or just by itself, but that's kinda lame. You could dig into the guts of tuple, which is actually just a compile-time set of cons cells, but that gets complex. Let's break out a little template metaprogramming and see where we get:

template<typename tuple_type, typename F, int Index, int Max>
struct foreach_tuple_impl {
    void operator()(tuple_type & t, F f) {
        f(boost::get<Index>(t), Index);
        foreach_tuple_impl<tuple_type, F, Index + 1, Max>()(t, f);
    }
};

template<typename tuple_type, typename F>
void foreach_tuple_element(tuple_type & t, F f)
{
    foreach_tuple_impl<
        tuple_type,
        F,
        0,
        boost::tuples::length<tuple_type>::value - 1
    >()(t, f);
}

Simple, right? Let's start at the bottom. foreach_tuple_element takes any old tuple and any old function as arguments. It then instantiates a foreach_tuple_impl with those arguments, as well as two additional template arguments. First, a 0, which is the index to start iterating at. Second, the length of the tuple minus one, which we'll get to in a second. foreach_tuple_impl calls f with the value at index Index using boost::get<Index>(t) and then recursively calls itself with Index + 1. Great! Done! Time for a beer and a bratwurst and a happy Memorial Day!

Compile that, though, and you'll notice a little problem. Namely that the compiler will never actually finish. It'll spin faster and faster, spewing an infinite stream of error messages to stderr. In order to actually stop the recursion you'll need to add one more functor:

template<typename tuple_type, typename F, int Max>
struct foreach_tuple_impl<tuple_type, F, Max, Max> {
    void operator()(tuple_type & t, F f) {
        f(boost::get<Max>(t), Max);
    }
};

This gets called when Index and Max are the same number and does not recurse. Now you can use foreach_tuple_element like so:

struct print_element
{
    template<typename T>
    void operator()(const T & t, const int index)
    {
        cout << index << ": " << t << endl;
    }
};

...

delicious_tuple strawberry(10, "chocolate", true);
foreach_tuple_element( strawberry, print_element() );

// prints this:
//    1: 10
//    2: chocolate
//    3: true

The fact that it's a recursive solution involving templates might scare some people off, but because tuples are guarateed to only have 10 elements it's pretty safe to say you're not going to blow the stack. Is there a better way to do this? Probably. This was a fun diversion to go with my Sunday morning bagel and coffee, though.

Note: this was inspired by a forum post in this German c++ forum but since I can't read German I had to puzzle it out and I thought I'd share.

Everyone Needs Goals

2010-05-27T20:37:00+00:00

Creating actionable information out of raw data is sometimes pretty simple, requiring only small changes. Of the few feature requests that I've received for Calorific, most (all) of them have been for goals. Always listen to the audience, that's my motto! With the latest version you can set up goals like this:

- goals: 
    - kcal:    2200
    - protein: [ 100, 200 ]
    
- 2010-05-27 breakfast:
    - 1000 kcal
    - 25 protein

- 2010-05-27 lunch:
    - 850 kcal
    - 50 protein
    
- 2010-05-27 lunch:
    - 500 kcal
    - 50 protein

This example is super simplified, of course, but you can see how it works. Creating an entry with the special name goals with one component for each nutrient you have a goal for. The value of each component is either a single number, which will be taken as a maximum, or a two element range.

Right now these are displayed by changing the color of the values on aggregate reports (daily and weekly). Red means "outside the range" and green means "inside the range".

$ calorific
2010-05-27 <total>                2350 kcal
                                   125 prot

Colors are done using Term::ANSIColor, which is included in core perl. Adding them was fairly easy, because of a simple function colored, which takes a scalar or an arrayref of scalars and a color argument and returns the scalar wrapped in the correct ANSI codes. Future display options could be displaying how much of each nutrient you have left for the day, maybe in a little progress bar type thing. Feature requests and comments are welcome, as always.

Program your Finances: Command-line Accounting

2010-05-23T15:15:00+00:00

Note: you can find much more information about ledger on ledger-cli.org, including links to official documentation and other implementations

About three years ago I was in some serious financial straits. I had just started my first job out of college that I had moved across the country for and had to bootstrap almost my whole life. This meant buying furniture, buying a car, outfitting a kitchen, etc. Every two weeks I would get a salary deposit, and within two weeks it would be almost completely gone from my checking account. I actually bounced a rent check or two in there. After the second time that happened I vowed it wouldn't happen again and started keeping track of every penny that I spent using a program called ledger. This was, in hindsight, exactly what I needed to get myself back on track. Actually seeing money moving in and out of my accounts forced me to modify my behavior. At the time, Mint wasn't around, but I don't think it would have helped nearly as much. Forcing myself to actually type out the transactions was the key to changing behavior.

Ledger is almost the most boring, austere accounting program you could think of. There's no pretty graphs, no online interaction, no GUI of any sort. It's basically a command-line driven calculator with a lot of specializations that make it ideal for tracking finances, which is what makes it so ideal for someone who spends a lot of time inside a text editor. It's very easy to script around and it has a very rich query language that lets you get at the data that you want with a minimum of fuss. It's very much the inspiration for Calorific.

The basic idea is that you write down all of your financial transactions in a text file with an easy-to-master syntax and then run the ledger command on them to generate reports. Here's a simplified extract from my ledger file:

2010/05/20 * Opening Balances
    Assets:Checking                          $500.00
    Liabilities:Amex                         $-10.00
    Equity

2010/05/21 * Salary
    Assets:Checking                        $1,000.00
    Expenses:Taxes:Federal                   $250.00
    Expenses:Taxes:State                     $100.00
    Expenses:Taxes:Social Security            $80.00
    Expenses:Insurance:Medical                $20.00
    Expenses:Insurance:Dental                  $2.00
    Income:Salary                         $-1,452.00

2010/05/21 Rent
    Expenses:Rent                            $600.00
    Assets:Checking

2010/05/21 Pacific Power
    Expenses:Utils:Electric                   $61.75
    Assets:Checking

2010/05/21 * AT&T Wireless
    Expenses:Cell Phone                       $88.46
    Assets:Checking

2010/05/22 NW Natural
    Expenses:Utils:Gas                        $20.31
    Assets:Checking

2010/05/22 Pizzicato
    Expenses:Food:Lunch                        $7.90
    Assets:Checking

2010/05/23 Comcast
    Expenses:Cable                            $60.00
    Liabilities:Amex

This is actually a complete ledger file (you can download it here) that illustrates a few key points. First, ledger is a double-entry accounting system. Every entry has at least one from and at least one to. Generally, the first line of the entry is where the money goes to, and it's a positive amount, with the second line being where the money comes from. If you leave off the amount of one of the lines ledger will automatically fill it in and make the entry balance. If you have an accounting background you can think of from and to in terms of debits and credits, but ledger doesn't force that. Second, accounts have a hierarchical namespace, which we can see like this:

$ ledger -f ledger.sample.txt -s bal
             $721.58  Assets:Checking
            $-490.00  Equity
           $1,290.42  Expenses
              $60.00    Cable
              $88.46    Cell Phone
               $7.90    Food:Lunch
              $22.00    Insurance
               $2.00      Dental
              $20.00      Medical
             $600.00    Rent
             $430.00    Taxes
             $250.00      Federal
              $80.00      Social Security
             $100.00      State
              $82.06    Utils
              $61.75      Electric
              $20.31      Gas
          $-1,452.00  Income:Salary
             $-70.00  Liabilities:Amex

This arrangement of accounts helps to maintain some sanity when dealing with lots of accounts, and it jives with the basic accounting equation: assets = liabilities + equity + (income - expenses). You'll notice that accounts just appear when you use them, sort of variables in perl without use strict;. This is both a blessing and a curse, because sometimes it's not obvious that you're misspelling things until you run reports and they look funny. The risk of messing up is mitigated if you use emacs by the bundled ledger.el major mode, which sets up tab completion for you.

Again using the example file, we can run some more detailed reports. For example, here's our checkbook register:

$ ledger -f ~/Documents/blog/static/ledger.sample.txt -r reg checking
2010/05/20 Opening Balances     Liabilities:Amex             $10.00       $10.00
                                Equity                      $490.00      $500.00
2010/05/21 Salary               Expenses:Taxes:Federal     $-250.00      $250.00
                                Expenses:Taxes:State       $-100.00      $150.00
                                Ex:Ta:Social Security       $-80.00       $70.00
                                Ex:Insurance:Medical        $-20.00       $50.00
                                Ex:Insurance:Dental          $-2.00       $48.00
                                Income:Salary             $1,452.00    $1,500.00
2010/05/21 Rent                 Expenses:Rent              $-600.00      $900.00
2010/05/21 Pacific Power        Ex:Utils:Electric           $-61.75      $838.25
2010/05/21 AT&T Wireless        Expenses:Cell Phone         $-88.46      $749.79
2010/05/22 NW Natural           Expenses:Utils:Gas          $-20.31      $729.48
2010/05/22 Pizzicato            Expenses:Food:Lunch          $-7.90      $721.58

Ledger will abbreviate account names as necessary when printing to make it fit in 80 columns. If you have a wider terminal you can pass the -w option to make it fit to 132 columns.

The power of ledger really comes into focus when you have more data available. One of the most interesting reports that I run gives me an idea of how I'm doing month-to-month by showing how much my assets have changed (negative numbers are better, in this case): ledger -MAn reg income expenses liabilities. The -M option groups transactions by month, -A will show the running average in the second column. By default it will show the running total. -n will group all transactions together, instead of showing one subtotal for each account. It's sort of boring with the sample file, though:

$ ledger -f ~/Documents/blog/static/ledger.sample.txt -MAn reg income expenses
2010/05/01 - 2010/05/23         <Total>                    $-161.58     $-161.58

In any of these examples you can change the output format to suit your needs. There are a lot of options here that are detailed in the manual (pdf), but here's one example. I have a little program in my bin directory called transpose, which takes three-column pipe-separated data and turns it into tab-separated values ready to be inserted into a spreadsheet. The first column is the row, the second column is the column, the third is the value to put in that cell. We can tell ledger to output, for example, a basic expense report formatted for transpose like this:

$ ledger -f ~/Documents/blog/static/ledger.sample.txt -F '%A|%D|%t\n' -M reg income expenses
Expenses:Cable|2010/05/01|$60.00
Expenses:Cell Phone|2010/05/01|$88.46
Expenses:Food:Lunch|2010/05/01|$7.90
Expenses:Insurance:Dental|2010/05/01|$2.00
Expenses:Insurance:Medical|2010/05/01|$20.00
Expenses:Rent|2010/05/01|$600.00
Expenses:Taxes:Federal|2010/05/01|$250.00
Expenses:Taxes:Social Security|2010/05/01|$80.00
Expenses:Taxes:State|2010/05/01|$100.00
Expenses:Utils:Electric|2010/05/01|$61.75
Expenses:Utils:Gas|2010/05/01|$20.31
Income:Salary|2010/05/01|$-1,452.00

With more data, this lets you easily compare month-to-month where you are spending money.

If you want to pull your financial life together but don't want to spend money on something like Quicken or trust Mint with your account credentials, I highly encourage you to try out ledger in addition to the other open source solutions like gnucash. All of these examples assume you're using version 2.6.2 of ledger, which you can download from the "Downloads" tab in github. Version 3.0 is just around the corner and it adds all kinds of neat things, including better automated transactions and a much more robust query language.

Building Battle Bots with Clojure

2010-05-16T23:00:00+00:00

Once in a while at Rentrak we have programming competitions, where anyone who wants to, including sysadmins and DBAs, can submit an entry for whatever the problem is. The previous contest involved writing a poker bot which had to play two-card hold'em, while others have involved problems similar in spirit to the Netflix Prize. This time we chose to build virtual robots that shoot each other with virtual cannons and go virtual boom! We'll be using RealTimeBattle, which is a piece of software designed specifically to facilitate contests of this sort. It's kind of like those other robot-battle systems, except instead of requiring you to write your robot in their own arbitrary, broken, horrible language, this lets you write your bot in any language that can talk on stdin and stdout.

Based on my previous entries the natural choice would be perl, right? I thought about it, actually. Started stubbing something out. Wrote some code to emulate enums and it worked on the first try, which brought to light the fact that I hadn't learned a new language in quite a long time and by not using a new language I was missing a golden opportunity. So, which language? The only real constraint that we, the Happy Fun Robot Times Killing Group, decided on was that it had to be easily installable on Ubuntu, which leaves the field pretty much wide open. Ruby? Already know it in passing. Python? Haven't done much with it for a few years but I don't think it's changed that much. Lisp? Hm. Intriging. Clojure looks interesting, and it's a good chance to figure out multithreading.

The RealTimeBattle system is conceptually pretty simple. Your robot is a little doughnut-shaped thing that can go forward, backward, accelerate, brake, and turn. In addition, it has a big cannon and a radar system, both of which can rotate independent of the bot itself. The radar is the only sensor you can rely on, although in some configurations you'll get coordinates relative to your start position every few game ticks.

When the game starts, the system will start up your bot in a child process and attach to stdin and stdout, so from the bot's point of view it's just talking a simple text protocol. In perl, talking this protocol would be a trivial combination of while(<>){ } and print, but in clojure it seems to be a might bit more complicated:

(loop []
  (let [in (read-line)]
    (if (not (nil? in))
      (do
        (println in)
        (recur)))))

Just writing that bit took me down about a dozen false starts, but I learned a whole lot about clojure in the process so I'm pretty sure it was worth it.

Ok, so now this little bot can listen, let's make it talk. RealTimeBattle has a command that your bot can send to the server to make it print out something in the message log. We can wrap that in a function like so:

(defn message [m & rest]
  (println (str "Print " m rest)))

and call that like this:

(message "Hi there my name is Botty McBotterson!")

The two other basic commands that I've implmented so far are Initialize, which will get sent when the system is ready to find out what name your bot has, and GameOption, which tells you all kinds of information about the environment that the bot lives in. Here's the whole program as it stands:

(def game-option-types [
  :robot_max_rotate
  :robot_cannon_max_rotate
  :robot_radar_max_rotate
  :robot_min_acceleration
  :robot_max_acceleration
  :robot_start_energy
  :robot_max_energy
  :robot_energy_levels
  :shot_speed
  :shot_min_energy
  :shot_max_energy
  :shot_energy_increase_speed
  :timeout
  :debug_level
  :send_robot_coordinates])

(def options (ref {}))

(defn message [m & rest]
  (println (str "Print " m rest)))

(defn robot-initialize [[first-round]]
  (if first-round
    (println "Name kabot")))

(defn robot-set-option
  [[option-number value]]
  (let [option-key (get
                    game-option-types
                    (Integer/parseInt option-number))
        option-val (Double/parseDouble value)]
    (dosync
     (alter options (fn [opts] (assoc opts option-key option-val))))
    (message (deref options))))
   
(defn process-input [m]
  (let [tokens        (seq (.split m " "))
        function-name (first tokens)
        args          (next tokens)]
    (message (str function-name " " args))
    (cond
      (= function-name "Initialize") (robot-initialize args)
      (= function-name "GameOption") (robot-set-option args)
      :else (message (str function-name " not implemented")))))

(loop []
  (let [in (read-line)]
    (if (not (nil? in))
      (do
        (process-input in)
        (recur)))))

This is pretty trivial at the moment. My basic design is to have the main thread deal with all of the I/O and updating a global state object, while another thread deals with analyzing this state and figuring out what to do. I haven't decided on any concrete strategies yet but for the first contest it'll probably be pretty stupid.

A few fun things to note: clojure provides very simple interop with Java classes and methods. For example, (.split m " ") calls the split method on m, which is actually just a Java String. The result of that is a String[], which isn't too useful in clojure so we immediately wrap it in a seq, which is sort of like a lazy cons list. Another example of this really trivial interop is the number parsing done in robot-set-option. I figured this out only after about an hour of thrashing about trying to figure out why passing a string as a vector index wasn't DWIMing like it does in perl. This is another example of why I need to do this project in another language. Perl has rotted my brain.

By the way, if there are things that I'm doing in this code that aren't idomatic clojure, please correct me. I just started learning today, after all. I found a pretty good tutorial which has guided me through basic types and stuff, but shortly I'll be branching beyond that into threading and agents and other fun things that it doesn't cover very well.

Actionable Information

2010-05-12T08:53:00+00:00

Let's pretend, just for a second, that you want to make some money on the stock market. Sounds easy, right? Buy low, sell high, yadda yadda blah blah blah. Except, how do you know when to buy and when to sell? Not so easy. Being a nerd, you want to teach your computer how to do this for you. But where to start? I discovered a few months ago that there are services out there that will sell you a data feed that literally blasts every single anonymous transaction that happens on any market in the US in real time. They'll also sell you access to a historical feed that provides the same tick-level information going back for several years.

So, ok, you've got a whole lot of raw data. All kinds of fun problems come from having a huge glug of raw data, especially when you're getting blasted more of it every day. Where to store it, how to store it, how to index it so you can get at certain segments quickly, etc. Let's pretend that you've solved all of those and it's time to get to the meat of this exercise: figuring out when to buy. You write a little program that searches through your historical data looking for signs of business cycles in various sized companies and gives you the top five that you should buy and how long you should probably hold onto them. That list is actionable information that you created out of raw data that you can use to make some money. Maybe. Hopefully. If the world doesn't end. Again.

Ok, that's a pretty small example. Let's do something bigger. Let's pretend that you're Google. You have truck loads of cash just laying around waiting for you to do something with it. Hey, data centers are cool, why not build some new ones! But where?

You know immediate things, like where your users are coming from and where the bottlenecks in your network are that prevent them from looking at your sweet sweet ads. Now remember, you're Google. You have a large chunk of all accumulated human knowledge at your fingertips. In addition to that stuff that every good company would know, you also know, somewhere deep down in your giant cache, things like where the zoning codes are favorable, where you have private fiber connections to and from, where you can get cheap electricity, voting patterns, histories of war riots and famine for every location on the planet. Lots of data. So, you write some Sawzall programs that go out and mine all this data and give you back likely locations, ranked by 10 year projected return on investment, and then you build at the top five places. Done. Easy.

In my admittedly limited time as a professional developer I've learned that probably close to 2/3 of my job is figuring out ways to suss out actionable information from vast quantities of low-level data. Be it displaying graphs or maps on a web page in the most understandable way, or trolling though a billion television remote clicks to determine who watched the Today Show this morning, it all boils down to providing some information to someone that they can act on.

In my personal life I need to have actionable information once in a while today. Before today Calorific could only tell exactly what I ate in its entirety or daily totals. That's somewhat useful, but sometimes I want to know what my weekly averages are, or limit the daily or detail reports to just a couple of days. To address those issues I added --begin and --end filters which will limit any report to just that day range. Specifying just one will leave the other as an open range. Calorific parses dates using DateTime::Format::Natural, which means it does the right thing with basically any date format you throw at, including relative dates like yesterday or 3 days ago . Also, I added a weekly report which prints daily averages for each week in the day range. This is the new default, which actually I'm not really sure about. Easy to change.

The next features on the docket are goals, which will let you set goal ranges for each base nutrient, an option to show you each day total or week average against the goal, and a summary report that shows you how much of each nutrient you've eaten today and how close you are to your goals. Stay tuned!

Moose vs Mouse and OOP in Perl

2010-05-09T08:00:00+00:00

After using Calorific for a month two things have become very clear. First, I need to eat less. Holy crap do I need to eat less. I went on to SparkPeople just to get an idea of what I should be eating, and it told me between 2300 and 2680 kcal. I haven't implemented averaging yet, but a little grep/awk magic tells me I'm averaging 2793 kcal per day. This is too much. So. One thing to work on.

Second, in the morning after I come back from lifting and sit down to enter my breakfast, I just add three lines to my calories file:

- 2010-05-07 breakfast:
    - 1 workout breakfast (blues)

and then type calorific in my shell, it takes ages to start up. Literally several seconds on a cold cache. I was pretty sure that this was due to the fact that I use Moose to help me define the four classes that compose Calorific. Now, Moose is great. Before writing Calorific I had only used a really old version of Class::MethodMaker or Class::Struct to build classes. That or build them myself, which is always fun (FUN FACT blessed array refs are wicked fast if you can get away with them). Moose is sort of a revelation. In the simplest case, you can say

package Foo::Bar;

use Moose;

has [qw/ baz blah frob /] => (is => 'rw');

1;

And you have yourself a fully functional class with three properties with read-write accessors. Pretty snazzy. However, you can get way more advanced:

package Calorific;

use Moose;

has 'filename' => (
    is       => 'ro',
    required => 1,
);

has 'recipes'  => (
    is      => 'ro',
    traits  => [ 'Hash' ],
    isa     => 'HashRef',
    lazy    => 1,
    default => sub { {} },
    handles => {
        getrecipe => 'get',
        setrecipe => 'set',
    },
);

has 'entries' => (
    is      => 'rw',
    traits  => [ 'Array' ],
    isa     => 'ArrayRef',
    lazy    => 1,
    default => sub { [] },
    handles => {
        addentries    => 'push',
        filterentries => 'grep',
        numentries    => 'count',
        allentries    => 'elements',
        sorted_entries => 'sort',
    },
);

1;

This is directly from Calorific. It defines three properties: a read-only simple scalar named filename which is required to be present in the call to new(), a recipes property which contains a hash ref and gets two accessors, get_recipe and set_recipe, which you call like this:

$calorificinstance->setrecipe('foo', 'bar');
$calorificinstance->getrecipe('foo'); # returns 'bar'

In addition, it sets up one more property named entries which contains an array ref and defines five accessors. There are actually more accessors defined than the code uses, but they're basically free so why not? You can see what they do and their calling conventions in the Moose::Meta::Attribute::Native::Trait::Array docs.

Ok, so Moose is great! Except, it's slow. Way slow. Wicked slow, especially on a groggy cache like my laptop has when I rudely wake it up in the morning and demand it actually do something for me for once. Geeze.

HOWEVER, there's a neat little project called Mouse, which has the lofty goal of emulating all of the sugar of Moose without any of the fat. Meaning, it doesn't pay nearly as large of a compile-time penalty that Moose does while retaining most of it's meta-y goodness. I ran one little command on the source tree yesterday evening and bam, just like that, everything was three times as fast.

find . -name '*.pm' | xargs perl -pi -e 's/Moose/Mouse/g'

Actually I had to install MouseX::NativeTraits from CPAN before everything worked but that's just details.

Anyway, the moral of the story is that Moose is great and makes building classes really easy and all, but if you care about startup speed and not so much about delving into meta classes and such, Mouse should be your go-to class. And in fact, you don't have to make that choice. There's another project called Any::Moose, which will load Mouse unless you declare you want Moose, which can be set with an environment variable. Pretty neat.

Calorific, a Simple Calorie Tracker

2010-04-08T19:00:00+00:00

I'm a nerd. I write software for a living. I spend a lot of my day either sitting in a chair in front of a computer, or laying on my couch using my laptop. I'm not what you'd call... athletic. I did start lifting weights about six months ago but that's really just led to gaining more weight, not losing it. A few years back I started counting calories and I lost some weight, and then stopped counting calories and gained it all back. Time to change that.

Now, I could use one of the many, many online calorie trackers. They're all ok and they have the advantage of being able to enter data whenever and where ever you are, but most of them have ads and using a web interface is kind of slow and staring at ads sucks. Also, the reports you can generate from them are always a bit limited. What if I want to see a monthly average of how many calories I ate as snacks? Or how many calories I shoved down my gullet from fast food? Or maybe I want to track another nutrient, like grams of protein. Doing all of this through a limited web interface would be tricky, to say the least. There has to be a better way.

I've been using this program called ledger for more than three years now to keep track of my finances. The idea is that you maintain a text file that contains all of your transactions in a really simple format, and then you can run basically arbitrary reports on it. I always have emacs open, so maintaining that file is a snap. I'd like to maintain my calorie history in the same way, using a lightly formatted text file. I actually tried to use ledger for this purpose but the syntax just wasn't right. What I really wanted was a way to build up foods from simpler foods, and have those be built from other, simpler foods, all the way down to calories. Something like this:

1 cup milk             = 100 kcal
1 scoop protein powder = 65 kcal
1 protein shake =
    1.5 cup milk,
    2 scoop protein powder
    
2010-04-08 breakfast
    1 protein shake

I danced around this format for quite awhile, trying to parse it line-wise and trying to parse it with Parse::RecDescent and treetop, and nothing ever really fit. Then, I punted. What's a lightweight, human readable format that already has a parser built? Why, YAML of course! Here's the same thing as a YAML snippet:

- 1 cup milk: 100 kcal
- 1 scoop protein powder: 65 kcal
- 1 protein shake:
    - 1.5 cup milk
    - 2 scoop protein powder
    
- 2010-04-08 breakfast:
    - 1 protein shake

The basic idea revolves around the concept of a recipe. Essentially, a recipe is a count, a label, and a bunch of components that can also be recipes. "100 kcal" is actually a recipe all by itself. Entries are just recipes that have a date instead of a count. At run-time, we resolve all the labels into recipes and then recursively get the values. Ideally everyhing will resolve down to a handful of base units, like "kcal" or "g protein", but if something doesn't resolve it'll get included right into the output.

So, ok, now I just need a program to analyze this stupid thing and print me some reports. That's where Calorific comes in. It's a little application (<500 lines, actually) that parses that YAML file and prints out either a detail or daily report. I have some big plans for it, including a report that gives the monthly average of daily totals, options to limit the date range you want to report, and 30 day moving averages. Installation instructions are in the readme file, if you'd like to try it out.

Adding RSS and Other Things

2010-03-29T20:34:00+00:00

Someone at work today demanded that I add an RSS feed, so here you go: atom. It didn't take very much to hack it in. Basically, all I had to do was install a few more CPAN modules, specifically DateTime::Format::Natural, DateTime::Format::W3CDTF, and finally XML::Atom::SimpleFeed. The first two are so I can put natural-looking dates in my entries and still be able to get full-fledged DateTime objects out of them, and the second is to save me the pain of writing out the Atom format's preferred datetime format. Also, I get neat date formatting in blog entires almost for free with the CLDR syntax.

Another thing to notice: File::Slurp instead of my own read_file_contents and write_file_contents. It works just as well as mine, except it's more sensitive to list vs scalar context.

Yet Another Static HTML Blog

2010-03-28T22:15:00+00:00

I'm a strict believer in learning by doing. It's how I learn best. In the spirit of learning, then, here's how I built the engine that powers this blog.

Right away I decided that there's no point in having a database to back this thing. The only useful thing that a database brings to the table is comments, and those are way more hassle than they're worth. Better to leave the comments at reddit or hacker news, where they already know how to deal with spam. Not having to worry about a database freed me up to worry about more important things, like how to put text on the screen. I'm most familiar with perl at the moment so I decided that the best way to build it would be a client-side script that generates some static html.

Current features:

Absolutely no database
Generates fully static html
Automatically ships it to my server
A really cheesy template system because I didn't want to learn Template::Toolkit just yet
Archives for everything, and only show the last 10 entries on the front page
Static pages (although currently there aren't any)
Markdown parsing for entries

If you want to see the source for it (including all the entries), it's on github, but I warn you it's kind of lame. The template system in particular is not really what I want it to be yet. It's non-recursive, so publish.pl basically acts like the top-level template. I'll probably end up converting it to Template::Toolkit at some point.