Sean Carley

How has Ruby blown my mind?

2007-02-02T07:44:00.000-08:00

In a word iterators. No single language feature has changed the way I think about code more than the simple, now seemingly obvious, pervasive, compelling act of passing anonymous code into a method to be run in context by that object. In a few days, this feature became so natural to me that languages where it was missing - Java, C, Perl - were harder just because I knew it was missing.

By the way, if you don't know about Pat Eyler's & Apress' ongoing blog about Ruby contests, check out this month's contest.

Install MIT Scheme

2006-09-07T14:58:00.000-07:00

Pat has convinced me to set some minor goals on the way to completing Structure and Interpretation of Computer Programs together. The first goal is to get the development environment installed. We have agreed to complete it by Monday, September 11th.

See more progress on: Read SICP

ActiveRecord, Delegation and Demeter

2006-09-01T20:47:00.000-07:00

Many of you have heard of the Law of Demeter (LoD). For those who haven't, I recommend reading Robert C. Martin's great article (in his Engineering Notebooks), the Dependency Inversion Principle.

There are many great descriptions of LoD online but in practice, it boils down to dot chains are usually bad. (E.G. foo.bar.baz) The reason they are bad is because they generally represent a place where encapsulation is broken. In my example, the currently running code knows that foo provides bar, which is ok, but it also knows that bar provides baz. Really, the currently running code should only know the methods on foo. If it needs baz, it should call foo.baz. Note that this is not the case with Java's sb = new StringBuffer(); sb.append(str).append(str); since in this example, you are working on sb each time and not descending down into sb's children.

The problem with trying to implement LoD in real code is that short of a complicated framework for avoiding the dot chains, you tend to end up writing lots of small delegation methods and you tend to have to write new ones every time a child's implementation changes. Obviously this becomes fragile over time and explains why there are refactorings for adding and removing delegation in Martin Fowler's famous book, Refactoring.

Fortunately, I don't code in average languages anymore. These days, I am all about Ruby. In particular, I am usually writing Ruby in the context of Rails. Over the last couple of months, my boss and I have developed an elegant way to add delegation to ActiveRecord objects. Our technique could be extrapolated to a more general case but ActiveRecord models is where we want it so that is all we have done.

The following example explains how our code used to work.

class User < ActiveRecord::Base; end
class Agent < User
  has_one :agent_detail
end
class AgentDetail < ActiveRecord::Base
  belongs_to :agent
  belongs_to :address
end
class Address < ActiveRecord::Base
  has_one :agent_detail
end

This is backed by a database with these tables:

users: id, type, *
agent_details: id, agent_id, address_id, *
address: id, *

So, agents are users (and exist in the users table) but since agent needs more fields than user does and we didn't want to pollute the users table with the needs of every kind of user, we provide an agent_detail table that contains all the information that agents need and users don't. Also, since addresses are a cross-cutting concern in an application, we decided to store all addresses in one table which can be referenced by others like agent_details.

So, say you have an agent bob. To get bob's zip code, you would have to do bob.agent_detail.address.zip_code, breaking encapsulation and delving far too deep into implementation details of an agent. What we really want to be able to do is bob.zip_code. Our code shouldn't care how bob implements this field but it should be able to get this information directly from bob. We also need to be able to set this information on bob with bob.zip_code = '12345'. The normal and naive way to do this follows:

class Agent < User
  has_one :agent_detail

  def zip_code
    agent_detail.zip_code
  end

  def zip_code=(zip_code)
    agent_detail.zip_code = zip_code
  end
end
class AgentDetail < ActiveRecord::Base
  belongs_to :agent
  belongs_to :address

  def zip_code
    address.zip_code
  end

  def zip_code=(zip_code)
    address.zip_code = zip_code
  end
end

This allows us to do bob.zip_code but it has some problems. For one, at the moment, nothing is ensuring all agents have an agent_detail and all agent_details have an address. This means we are subject to nil problems (also know as NullPointerExceptions in the Java world). Also, say we wanted to change the name of zip_code to zip? We would have to change a table and a total of 8 lines of code in 2 different classes. Not exactly the most robust system. This also has some problems with saving dependent objects. If you do bob.zip = '12345'; bob.save, odds are the you expect the zip code change on address to be saved but it isn't.

Ruby and Rails have some great tools that can help us simplify the code above and remove some of the fragility of the interface in the face of change. Specifically, method_missing comes to the rescue to handle the bob.zip_code method and delegate it down and before_save and after_safe ActiveRecord callbacks help solve the save problems. This makes your code look like this:

class Agent < User
  has_one :agent_detail

  after_save {|agent| agent.agent_detail.save}

  def safe_agent_detail
    agent_detail.nil? : build_agent_detail ? agent_detail
  end

  def method_missing(method_id, *args)
    if safe_agent_detail.respond_to? method_id
      agent_detail.send(method_id, *args)
    else
      super
    end
  end

  def respond_to?(method_id)
    safe_agent_detail.respond_to?(method_id) || super
  end
end
class AgentDetail < ActiveRecord::Base
  belongs_to :agent
  belongs_to :address

  before_save {|agent_detail| agent_detail.address.save}

  def safe_address
    address.nil? : build_address ? address
  end

  def method_missing(method_id, *args)
    if safe_address.respond_to? method_id
      address.send(method_id, *args)
    else
      super
    end
  end

  def respond_to?(method_id)
    safe_address.respond_to?(method_id) || super
  end
end

This does neatly solve most of our problems but there is still one nagging itch, this code isn't very DRY. Most of the details in AgentDetail and Address are the same. What if we could take a page out of Rails and do this:

class User < ActiveRecord::Base; end
class Agent < User
  has_one_delegate :agent_detail
end
class AgentDetail < ActiveRecord::Base
  belongs_to :agent
  belongs_to_delegate :address
end
class Address < ActiveRecord::Base
  has_one :agent_detail
end

This is in fact, what my boss and I have done. The following code provides has_one_delegate and belongs_to_delegate:

module ActiveRecord
  class Base
    def self.belongs_to_delegate(delegate_id, options = {})
      delegate_to(:belongs_to, delegate_id, options)
    end

    def self.has_one_delegate(delegate_id, options = {})
      delegate_to(:has_one, delegate_id, options)
    end

    private

    def self.delegate_to(macro, delegate_id, options = {})
      send macro, delegate_id, options

      save_callback = {:belongs_to => :before_save, :has_one => :after_save}[macro]

      send save_callback do |model|
        model.send(delegate_id).save
      end

      delegate_names = respond_to?('safe_delegate_names') ? safe_delegate_names : []
      delegate_names = (delegate_names - [delegate_id]) + [delegate_id]

      def_string, source_file, source_line = <<-"end_eval", __FILE__, __LINE__

        def self.safe_delegate_names
          #{delegate_names.inspect}
        end

        def safe_delegates
          self.class.safe_delegate_names.collect do |delegate_name|
            send(delegate_name).nil? ? send("build_\#{delegate_name}") : send(delegate_name)
          end
        end

        def method_missing(method_id, *args)
          safe_delegates.each do |delegate|
            return delegate.send(method_id, *args) if delegate.respond_to?(method_id)
          end
          super
        end

        def respond_to?(*args)
          safe_delegates.any? {|delegate| delegate.respond_to?(*args)} || super
        end

      end_eval

      module_eval def_string, source_file, source_line + 1
    end
  end
end

The code itself isn't the most beautiful internally, but the simplicity it provides is elegant and effective. Also, because we eventually did need more than one delegate for the same class, it is robust enough to allow multiple delegates. Score one for the Ruby/Rails team. An entire class of refactorings goes from recipe to method call. This is why I love programming.

Eventually, we plan to release this as a plugin to the Rails community. Until then, let me know what you think. Would you use it?

When Inheritance Doesn't Pay - Problems Testing Rails

2006-06-21T12:00:00.000-07:00

The smallest things have the capacity to cause the most pain. Today's pain is caused by 5 characters, or to be more acurate, the absence of 5 characters.

super

Say it with me, super. Yep, super. Guess what Rails doesn't include in the setup of functional tests by default. You betcha, super. And the funny thing is, most of the time, it really doesn't matter. But sometimes, oh boy, yeah, sometimes it makes all the difference.

Given two test classes:

a_test.rb

require File.dirname(__FILE__) + '/../test_helper'

class ATest < Test::Unit::TestCase
  fixtures :empties

  def setup
  end

  def test_fixture
    empties(:first)
  end
end

b_test.rb

require File.dirname(__FILE__) + '/a_test'
class BTest < ATest; end

Would it surprise you that run seperately, both of these pass with flying colors, but run together, they result in abominable crashes? It sure surprised me.

[~/projects/test]$ /usr/bin/ruby -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" "test/unit/a_test.rb" Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
Started
.
Finished in 0.115591 seconds.

1 tests, 0 assertions, 0 failures, 0 errors
[~/projects/test]$ /usr/bin/ruby -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" "test/unit/b_test.rb" Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
Started
..
Finished in 0.157902 seconds.

2 tests, 0 assertions, 0 failures, 0 errors
[~/projects/test]$ /usr/bin/ruby -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" "test/unit/a_test.rb" "test/unit/b_test.rb"
Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
Started
EEEE
Finished in 0.088633 seconds.

  1) Error:
test_fixture(ATest):
NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occured while evaluating nil.[]
    /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/fixtures.rb:475:in `empties'
    ./test/unit/a_test.rb:10:in `test_fixture'

  2) Error:
test_fixture(ATest):
NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occured while evaluating nil.-
    /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/transactions.rb:112:in `unlock_mutex'
    /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/fixtures.rb:534:in `teardown'

  3) Error:
test_fixture(BTest):
NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occured while evaluating nil.[]
    /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/fixtures.rb:475:in `empties'
    ./test/unit/a_test.rb:10:in `test_fixture'

  4) Error:
test_fixture(BTest):
NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occured while evaluating nil.-
    /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/transactions.rb:112:in `unlock_mutex'
    /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/fixtures.rb:534:in `teardown'

2 tests, 0 assertions, 0 failures, 4 errors
[~/projects/test]$

The key to this dilemma should be pretty easy to find now that I have bloodied myself on the problem for you. Exactly, super. Add super to the setup and all our woes are cured. I haven't got the slightest idea what magic TestCase.setup is doing but without it, we bomb every time. Actually, I suspect inheritance isn't even the culprit here. Running a with itself twice exhibits the same behavior, with the same solution.

I suspect the error condition is related to the edge case of a test class completing then being loaded again but I'm not sure how to prove it. I'm guessing the 2nd time through, an action isn't take because Rails is convinced it already happened. But the moral is clear, super is your friend.

Compounding Errors - Another Rails Testing Adventure

2006-06-07T11:07:00.001-07:00

The Moral:
Always include test_helper in Rails tests.
Be specific when handling exceptions.
Use diff. Trust diff.
Pair often. Pair always.

The Story:
Every test class generated by Rails includes require File.dirname(__FILE__) + '/../test_helper'. If you remove it, or you write your own test and forget it, bad things can happen, things like this....

First things first, how are the tests? rake test_functional. Failure. Hmm. I expected that. This project was written before we fully committed to test first. It shouldn't be too hard to get the existing tests to run. Let's see what we need.

The failure is in the functional tests. A require is failing and I don't know why. I wonder where that file is supposed to come from? Maybe it is really missing. There is something similar in another one of our projects. For now, I will just include the mock we used to make that test pass. (Copy the test/mock directory from one rails app to the other.)

That should fix it. rake test_functional. Blamo. Still failing. Hmm. Naming problems? File names seem to match. Hack. Hack. Hack. This sucks, I've burned an afternoon and there's nothing to show for it.

Interlude. A night of little sleep and a nagging weight on my mind. Not fun. The tension wakes me up at 3:00 AM and I start hacking from home. It isn't like I was getting any useful rest anyway.

So, the test isn't finding my mock either. I don't get it. Well, I guess I can check my load path before the require.

puts $:.inspect
 ["lib", "test", "/usr/lib/site_ruby/1.8", "/usr/lib/site_ruby/1.8/i386-linux",
 "/usr/lib/site_ruby", "/usr/lib/ruby/1.8", "/usr/lib/ruby/1.8/i386-linux",
 "."]

That doesn't look right. None of my rails includes are there. Hack. Hack. Hack.
Burned several morning hours and I still have to face Steve and tell him I have nothing to show for it. Having that hanging over my head sure does make for a long morning commute.

OK. Before I talk to Steve about this, lets at least check his machine. rake test_functional Passed. Everything passed. Wait. WTF do you mean everything passed?! OK. Maybe I'm just an idiot (I mean past experience suggest this must frequently be the case.) so lets make sure I have the current code. svn status. Nothing modified. svn udpate. Nothing missing. Well, maybe Steve made changes on his machine. svn status. Nothing modified. svn udpate. Nothing missing.

I've been here before. Code works on one machine but not on the other. The problem is, this time, it is well tested code and I can't find anything that is different. Discussing it with Steve didn't help either. He spent a chunk of time getting all these tests to pass just recently so they should still work.

Well, my machine is different. We are planning a system upgrade and I'm the guinea pig. But most of our software is the same. Sure, the OS is different right now but.... Well, I can at least standardize my software install. I'll just get our environment setup script to work.

Great. That script blows up on my machine too. It can't get the file from subversion. Well, of course it can't. That doesn't look like the repository location to me. Hack, hack. Oh. Idiot. That was right. The subversion repository is being served up by http and I was looking for the svn+ssh location.

Well, it still blows up. Compare the file path. Still missing. Load the directory in a browser. There the file is. But, umm, well. OK. Maybe it is something in the version. That looks different in the script. Nope. That looks right.

Eventually, I notice the file name the script is looking for is .tar.bz2 but the file name in the directory is .tar.gz. Don't ask how long it took. Too long.

It's looking for the wrong file name. Why. Where does that come from. There it is. But, wait, it should be looking for the name that is there first. If it doesn't find that then it catches the exception and tries the other name.

The light bulb finally goes on.

Ah. I'm getting an exception other than the expected exception and that is incorrectly being understood as a missing file. And hey, while I'm at it, guess what, I should be running this as root and I'm not. The real error is a permissions error but I can't see that because we handled every exception instead of just the missing file exception we expected. And surprise, surprise, everything works great when I run it as root. I leave the script to run over night since it is taking too long.

Interlude. Another bad night. Another early morning. This time I was so burned out that instead of going to hacking night, I crashed for 3 hours when I got home. Of course that meant I couldn't get back to sleep until the wee hours. And of course that didn't stop me from waking up way too early.

Back at the office again. Everything is good on Steve's machine, and not good on my machine. Well, lets diff the output. I should have done this days ago but hindsight is 20/20 and foresight is blind as a bat. Well, the commands for the tests aren't in the same order but I don't really see anything else. Lets follow the rake trace.

The trace is different but I don't really understand why. Hack. Hack. Oh, by the way, things work in autotest but not rake. Now that is just creepy.

Finally, I corner Steve to help me out. I'm not solving this alone, so lets try some of that pairing juju.

Hack here. Hack there. Trace this. Trace that. Eventually Steve notices what I already knew but this time, Steve gets it when I just blanked. It is order dependent. The tests are being run in a different order. Hack, hack. Well, we can't really control the order, it seems to be an artifact of the different OS. Still, we should be able to figure out what causes the order dependency. Hack. Hack.

Bingo, there it is. This test (which was copied from another, non-rails project) doesn't include test_helper. And guess what. test_helper is where environment.rb is loaded. And environment.rb is where the additional include paths are added, not to mention that test_helper is where the mock paths are added. No wonder it couldn't find those mocks that I added.

Finally, we get to here and now. Two and a half days of developer time wasted over several small errors, compounded together. Cory Foy recently wrote "And that's the trick with being Agile. If you slip, even for a second, it *will* bite you. You have to stay on guard constantly". This is true of Agile but really, it is true of all development. How many times have you been bitten by small problems that take days to figure out and would have been avoided with just that little bit more effort?

Of course, the mark of a great developer isn't the lack of mistakes. The mark of a great developer is learning from mistakes and preventing them in the future. Soon, we will either have something that tests to make sure our tests include test_helper, or even better a way to avoid that step in every single bloody test file. We will also be more careful to trap only what we expect and leave the unexpected unhandled or at least properly handled. And I will definitely learn to trust my diffs earlier.

Don't make my mistakes, but more importantly, don't make your mistakes twice.

Assignment in Ruby - Block Assignment

2006-05-25T02:37:00.000-07:00

For the second installment of Assignment in Ruby, I would like to explore block assignment. Block assignment is very similar to the simple scoped assignments I explored in my first post. To understand the differences, it is necessary to understand how block variables are scoped.

When a block uses a local variable, Ruby has to determine what is being used. { |x, y, z| puts x, y, z } uses three local variables. Interestingly, these three variables could each potentially have a different scope.

If there is an x in the current local scope, the variable x used in the block will be that local variable, meaning the assignment will be an :lasgn (local assignment).

If there is no y in the current local scope but there is a y in an enclosing block, the y variable used will be the block scoped variable from the outer block and the assignment will be a :dasgn (block assignment).

If there is no z in the current local scope and there is no z in any enclosing blocks, then a new z will be created in the scope of the current block and the assignment will be a :dasgn_curr (block local assignment).

The different scopes are each illustrated in the following code:

$ echo "x = 123; foo() {|x| y = 456; bar() {|x,y,z| puts x, y, z}}" | parse_tree_show -f
[[:lasgn, :x, [:lit, 123]],
 [:iter,
  [:fcall, :foo],
  [:lasgn, :x],
  [:block,
   [:dasgn_curr, :y],
   [:dasgn_curr, :y, [:lit, 456]],
   [:iter,
    [:fcall, :bar],
    [:masgn, [:array, [:lasgn, :x], [:dasgn, :y], [:dasgn_curr, :z]]],
    [:fcall, :puts, [:array, [:lvar, :x], [:dvar, :y], [:dvar, :z]]]]]]]

Written a little more nicely, the code under inspection is:

x = 123
foo() { |x|
  y = 456
  bar() { |x,y,z|
    puts x, y, z
  }
}

Note the 2nd to last line of the S expression [:masgn, [:array, [:lasgn, :x], [:dasgn, :y], [:dasgn_curr, :z]]]. The first thing to notice is:masgn. :masgn stands for multiple assignment and will be covered in its own post. After the :masgn you can see concrete examples of the local assignment to x, the outer block assignment to y and the current block assignment to z.

This can be a sticky situation and I know I missed it when I was learning Ruby. I only internalized how block scopes are handled when I saw it reflected in ParseTree and the AST. I still don't really understand why you would want to reuse local scoped variables inside a block but that is probably an indication of my weakness with closures. If you have any ideas, please let me know.

I hope you have enjoyed another brief forray into Assignment in Ruby. Join me next time as we explore multiple assignment and how it can simplify your life.

Climbing without a Net

2006-05-25T02:17:00.000-07:00

Developing test first is like rock climbing on lead.

When a climber climbs lead, they have to place protection as they go. At any point, if they fall, their last piece of protection generally keeps them from falling far. Sometimes, that protection will fail and when it does, the other protection they have placed keeps them from death or serious injury.

Each test written is like a piece of protection we have placed. Usually, the last test we wrote is the most important but all the tests together keep us from serious injury. Our tests really are our safety net and I can't understand why people choose to live without it.

If test first is lead climbing, then it goes to reason that the test first challenge is top roping and developing without tests is free climbing. Sure the best climbers can skate up easy climbs without protection but for the rest of us, that is the path to certain doom.

When Rails Needs a Clue - Single Table Inheritance Problems

2006-05-22T09:41:00.000-07:00

Given Grandchild < Child < Parent < ActiveRecord::Base, your controllers need model :grandchild. Without it, well, this could happen:

There is a sinking feeling in your gut. You've been working in rails for weeks. You've been doing everything right, Test Driven Development to the hilt, when suddenly, the worst happens. Behavior in test doesn't match behavior in development. You begin to question whether any of your testing is valid. Does your app really work, or does it just seem to work in test? How can test and development possibly differ? Your worst nightmares are coming true.

This might seem like a ridiculous scenario but it is exactly what I have dealt with for the last week. We have a table with single table inheritance. Admin < Broker < User < ActiveRecord::Base all in table :users. Errors started showing up in development that passed in test. We tracked it down to differing sql statements. In test, Broker.find_all generated SELECT * FROM users WHERE ( (users.`type` = 'Broker' OR users.`type` = 'Admin' ) ). In development, the exact same Broker.find_all generated SELECT * FROM users WHERE ( (users.`type` = 'Broker' ) ).

We had no idea what was going on and we had features that had to be ready right now for a demo. We worked around the problem. We wrote our own find conditions and moved on, writing a todo to find out what was wrong. The sleepless nights started and the worry began to grow.

Over the weekend, the other developer on my team looked at some code I wrote Friday and had problems. He didn't have time to look into it but told me about it this morning. My first response, well, I'll write a test that shows the problem, only every test I wrote worked fine in test. I could reproduce his problems in development every time. This shouldn't be so hard.

Start from the basics. I created a test with exactly the same input as was reported in the error on development. Same parameters. Same session. Same flash. Everything was exactly the same. Only, what failed in development passed in test. The sick feeling in my stomach is getting stronger. I feel like I'm in a dream watching my career go down the drain. Sure enough, I track it down to nearly the same problem as before. We have Default < ActiveRecord::Base; belongs_to Broker. If the default.broker_id pointed to a user with type=Admin, it failed in development. Default.broker was returning nil when it should have been returning an Admin. Again, the SQL was different between test and development in exactly the same way. Development would only accept Brokers and test accepted Brokers and Admins. This didn't make any sense and this time, we had time to look at it.

My partner started looking at the source in Rails and I started searching online. I didn't find anything with a few quick searches. I knew the #rubyonrails channel on Freenode is pretty active so I decided to ask my question there. "Has anyone seen differences between test and development when..." Hmm. I want the exact SQL I'm seeing in test and development before I ask this question. I know, I'll fire up the console for both and get it from the logs that way.

Loki took pity on me here and Coyote found something else to toy with. export RAILS_ENV=test; ./script/console. So far, so good. Broker.find_all; I only get Brokers back. WTF! That's the same thing I've been seeing in development. A quick check. Yes, I'm in test. Hmm. User.find_all; I get everything. (select * from users, no where clause) Admin.find_all; I get only Admins. I expected that, Admin doesn't have any children. Broker.find_all; Holy hand grenades. That is everything, Admins included. The where clause changed. Umm. But. Maybe this time I put something different. Just to be sure, I used up arrow to run the first Broker.find_all; (I couldn't see a difference but who knows at this point.) Sure enough, the original search turns up the new results. Then, wham, lightning struck. If Admin has never been loaded, Rails doesn't know Admin < Broker!

A quick conversation with my partner, a little test (added model :admin to the controller) and sure enough, everything is peachy. There is just one nagging doubt. I don't know how to create this in test. You see, single table inheritance is implemented in a single YAML fixture. When fixtures :users runs, Rails figures out I have a User, Broker, Customer (< User) and Admin model. Loading my fixture seems to make Rails aware of those relationships. So, how do I get data for admins into the table for users without making Rails aware of the Admin class? I have no idea.

The sinking feeling isn't gone but it is a good deal smaller. I now have a picture of the world that explains the unexplainable and test and development work the same again. But still, that little doubt.

If you know how to test this, won't you please let me in on the secret?

Lost but not Forgotten

2006-05-19T09:24:00.000-07:00

I still owe you several posts on the 12 kinds of Ruby assignment. I also have more to write about my sagas with ParseTree and the like. I can't promise to have them soon but I do promise to have them, eventually. In the meantime, consider this my formal apology for letting you down.

flunk, my autotest friend

2006-05-19T09:09:00.000-07:00

You use autotest, right? (gem install ZenTest and get on it if you don't). Well, then you know autotest auto-focuses on failing tests. You probably also love this and rely on it all the time. Well, leverage flunk and take your relationship with autotest to the next level.

test_some_silly_thing
  blah, blah, blah, blah
  this is a test that should be much smaller but isn't
  or is mystifying me with its strange behavior or something
  and I want to debug what the F@#! is going on, and
  I don't want to run the other 4800 tests that take more
  than the 8 milliseconds attention span I am willing to
  wait for my tests to run or, ... you get the idea
end

Now, lets make autotest focus on test_some_silly_thing. Add the following line as the last line of the test:


  flunk "Passed!"

Cool! Now, even if our test passes, we fail. This means autotest will stay focused while we play. You can now do things like check the database after your test runs (because it is the only test that was run and no other tests had a chance to muck things up).

By the way, if you don't already know, flunk is the equivalent of assert false. It is so much easier to read and understand than assert false is though.

A few links you might find useful:
ZenTest
Using autotest with Rails
article on ZenTest in Linux Journal
Dr. Brain rails against his own creation
Everything else Dr. Brain said about ZenTest in his blog

Assignment in Ruby - Simple Scoped Assignment

2006-04-12T18:38:00.000-07:00

The largest class of assignments in Ruby are the simple scoped assignments. Ruby represents assignment to a name in a particular scope with its own scope assignment node. I won't go into what nodes mean to the VM in this article but think of them as named expressions. The following table shows the simple scoped assignments and their related code and abstract syntax trees (AST).

Scope	AST Name	Code	AST
Local	`:lasgn`	`a=nil`	`[:lasgn, :a, [:nil]]`
Instance	`:iasgn`	`@a=nil`	`[:iasgn, :@a, [:nil]]`
Class	`:cvasgn`	`@@a=nil`	`[:cvasgn, :@@a, [:nil]]`
Global	`:gasgn`	`$a=nil`	`[:gasgn, :$a, [:nil]]`
Each of these ASTs was generated using ParseTree 1.4.1 and the following command: `echo "code" \| parse_tree_show -f`

As you can see, each of these has the same structure in the AST. By using different types of nodes for each one, it is simpler for the Ruby VM to determine where to look for a variable and set it.

The code below is a straight through SexpProcessor using the ParseTree gem and the included SexpProcessor as a base. This processor doesn't do anything exciting but by explicitly writing the process_type methods, we have exposed interaction points where we could do things like store a list of assigned variables, output funny messages or generate metrics.

Sexp stands for S-expression. ParseTree represents S-expressions in ruby as nested arrays of arrays. S-expressions are particularly well known for their use in most Lisp like languages to represent code and data. If you would like to know more about S-expressions, Google is your friend.

SexpProcessors have a few rules that you must observe in order to get a correct traversal of the expression. First, you need to understand that the dispatch method process(exp) is initially called for every pair of matched brackets. This is important because in our process_foo methods, we need to call process() on any members of exp that are arrays. If we don't, we essentially prune that piece of the S-expression from our processing. Usually, that is wrong. It is also important to not that process(exp) only takes arrays as input. If you call process(exp.shift) and the shifted element is a literal, you will have an error on your hands. process(exp) will then pass control onto process_foo(exp) where foo is the :literal that appears as the first element of the expression. (In our processor, because of the auto_shift_type in initialize, the first element of exp is shifted off before control passes to process_foo. This results in slightly cleaner, easier to read code in process foo. Instead of s(exp.shift, exp.shift, process(exp.shift)), we get the slightly more clear s(:iasgn, exp.shift, process(exp.shift)). This is obviously a personal preference.)

Another basic rule of SexpProcessor is that what comes in should be what comes out, or, in the case of auto_shift_type, what would have come out if auto_shift_type were false. This is best illustrated with an example.

process([:lasgn, :a, [nil]]) should return [:lasgn, :a, [nil]]

with auto_shift_type = false
process([:lasgn, :a, [nil]]) calls process_lasgn([:lasgn, :a, [nil]])

with auto_shift_type = true
process([:lasgn, :a, [nil]]) calls process_lasgn([:a, [nil]])

in either case process_lasgn() must return [:lasgn, :a, [nil]].

Still another rule, in a process_foo(exp) method, exp should be empty before the method returns. Failure to empty the exp is considered bad form (and is usually wrong too) and therefor raises an error. This is the quickest way to catch errors in coding like

# process_lasgn([:a, [nil]])
def process_lasgn(exp) 
  s(:lasgn, # auto_shift_type = true or this would be s(exp.shift,
    exp.shift) # :a
  # exp now == [[nil]]
end

This will really help you when you get the structure wrong for the s-expression representing a node.

The final bit of magic to understand before venturing off to write your own processor is s(). s(*args) is shorthand for Sexp.new(*args). It was added to keep things easier to read and as a user of ParseTree & SexpProcessor, I am sure you will appreciate it.

# A Straight Through SexpProcessor exposing
# the process_foo simple assignment methods.

begin require 'rubygems' rescue LoadError end
require 'parse_tree'
require 'sexp_processor'

class PassThroughProcessor < SexpProcessor
  def initialize
    super
    self.auto_shift_type = true
  end

  def process_lasgn(exp)
    s(:lasgn, exp.shift, process(exp.shift))
  end

  def process_iasgn(exp)
    s(:iasgn, exp.shift, process(exp.shift))
  end

  def process_cvasgn(exp)
    s(:cvasgn, exp.shift, process(exp.shift))
  end

  def process_gasgn(exp)
    s(:gasgn, exp.shift, process(exp.shift))
  end
end

You would invoke this processor on an unsuspecting class with the following magic incantation.

PassThroughProcessor.new.process(ParseTree.new.parse_tree(klass))

If you are a glutton for punishment, you could run it against PassThroughProcessor. (*Note: As is, this wouldn't create any output at all but the skeleton is all there. Enjoy yourself.*)

I have more than used up the time and space allotted for this article. I hope this was educational and I look forward to seeing you next time with Assignment in Ruby.

Assignment in Ruby

2006-04-10T04:09:00.000-07:00

Would you believe Ruby has a dozen different ways to represent assignment internally? This may be something you don't expect to find when you go poking around the internals of things like the Ruby implementation or ParseTree. I know I didn't.

Over the coming weeks, I will:

document the 12 different representations
discuss what they are and what is unique about them
show you what ruby code creates each one and the AST (abstract syntax tree) generated by ParseTree for each
show you a code example of how you would write your own parse_foo method in a SexpParser if you wanted to track something about each one
show you what we have done (if anything) with each one in CheckR and explain why

Join me in an journey under the covers as I explore assignment in Ruby.

Ping Pong Programming - No Pairing

2006-04-08T23:51:00.000-07:00

The paradigm of ping pong pairing can be used without pairing. I have been ping pong programming remotely with Pat Eyler for a couple weeks now and it has been tremendously effective. Basically, the process for ping ponging remotely is the same as ping pong pairing but instead of passing a keyboard back and forth, we check our code into Subversion. A typical scenario looks something like this:

Pat updates his code to make sure everything is current. He runs the test suite to see what the currently failing test is. After some analysis, he fixes the code and makes the test pass. Now, he checks his code into Subversion. Next, he writes a new test and confirms that it fails. Because he has to communicate the change to me, he checks his failing test into Subversion with a comment about the current test. Finally, he notifies me that the ball is in my court. It is my turn to repeat the process.

This style of development is more powerful than simple Test Driven Development for the same reasons I mentioned before. By posing tests for your opponent, you develop stronger tests than you do for yourself.

Remote ping pong does have its shortcomings though. In particular, it doesn’t deal with time imbalances very well. While Pat is implementing my test, I am doing nothing. If I have several hours to devote to coding and Pat only has a few minutes, I will spend a lot of time waiting for a test when I could be coding.

I want to address the time disparity concern by figuring out what works in ping pong and creating a model with all the benefits but without the time constrained challenges. In addition, I would like to figure out how to apply the principles of ping pong to teams of more than two people. A tenative name for this model of development is Test Exchange.

I will describe Test Exchange in a later post.

See more progress on: Create a form of ping pong programming for distributed teams.

Ping Pong Pair Programming

2006-04-08T23:26:00.000-07:00

I don’t know who first coined the name Ping Pong Pair Programming but I know from the first time I heard of it, I was convinced. It seems likely that PPP (ping pong pairing) arose out of a need to address some of the challenges of pair programming. Namely, the difficulty of maintaining attention when you are not the one at the keyboard.

PPP is also a derivative of Test Driven Development. In TDD, you write your test first and you never write code for a feature that is not in a failing test already. It is a great way to ensure a robust test suite for all of your development. TDD works like this: 1) write a test, 2) confirm that it fails, 3) write code to make the test pass, 4) confirm the test passes, 5) confirm no other tests are broken, 6) refactor mercilessly. TDD is very effective but there are challenges. It is very easy to shortcut your tests a little by writing more than you really need. It is also easy to avoid testing the really hard cases.

Ping pong pairing works with two people. Let us call them Bob and Tom. The day starts with Bob at the keyboard. All the tests pass. Bob writes a test. Bob confirms it fails. The keyboard passes to Tom. Tom works on making the test pass with Bob at his side, helping him along. Tom finishes his implementation and confirms all the tests pass. Tom writes a new test and confirms it fails. The keyboard returns to Bob.

PPP is very similar to TDD in that it is made up of te same steps. The difference is that after a failing test is implemented, the other developer takes control. This can develop into a friendly game. Bob tries to write a good test that forces Tom to implement real and valuable functionality. Tom tries to pass Bob’s test with the simplest thing that could possibly work. The harder Bob and Tom try to push the difficult work to the other person, the more robust the test suite and the finished code become.

Ping pong pairing addresses the psychological challenges in both pair programming and test driven development. Because control switches so frequently, partners stay more engaged in what is going on. Because you don’t have to implement the code for your own tests, you are much less likely to take shortcuts just to allow yourself to get by.

Ping pong pairing is hard work and is has been very rewarding when I have done it. It results in much better code and more robust tests. It also helps to keep things moving at a good pace. If you pair program, I strongly recommend you try your hand at PPP.

See more progress on: Create a form of ping pong programming for distributed teams.

StL.rb Hacking Nights

2006-04-08T04:25:00.000-07:00

Seattle.rb has a meeting every week. Once a month they have presentations and the other weeks are dedicated to hacking nights. I have envied this for quite a while and I finally decided to do something about it.

Tuesday, April 4 was the first ever StL.rb hacking night. After announcing it a couple weeks in advance, I spent 3 hours at Borders and 2 other members of StL.rb made it. Borders wasn’t the best place to hold the meetings so next week we are going to try St. Louis Bread Company. Most of you might know it as Panera Bread Company. They didn’t change the name in St. Louis. I guess people might have gotten upset.

I’ve made myself available for hacking any night people want but the weekly hacking nights are going to be on the same weeknight as our regular meeting. Since that is Tuesday, I will be too busy to be jealous of Seattle.rb’s meetings on the same night.

See more progress on: help build local Ruby Brigades

CheckR

2006-04-08T04:16:00.000-07:00

I started CheckR with a friend. It hopes to be a static source analysis tool for Ruby code, along the lines of lint for C/C++. We are actively developing on it in a remote, ping pong, test driven development way. We are very close to our first alpha release. The first release will hopefully add some small value and be able to catch a few errors earlier in the development cycle. I guess we need autocheckr too. And we definitely need to run checkr against itself.

See more progress on: start an open source project

Emacs + ZenTest = ZenMacs?

2006-04-06T12:04:00.000-07:00

Good friends make you a better person. Because of the recent example of Ryan Davis, Eric Hodel and especially Pat Eyler, and the less recent example of Gus Mueller, I am a better programmer who does better things. The following is one of those things.

I have been learning Emacs and the ZenTest suite at the same time. This morning I came up with the following bit of goodness.

M-x autotest from a buffer whose current dir is the root of your project results in:

You now have a shell with autotest running in the bottom left pane, an eshell ready for your subversion commands in the bottom right pane and you can work with your project in the top pane.

This is due to a macro I wrote to create this setup for me automagically. The macro does the following: (*warning emacs speak*)

C-x 2, C-x o, C-x 3
M-x shell
  autotest
M-x set-variable
  comint-scroll-to-bottom-on-output
  all
C-x o
M-x eshell
M-x set-variable
  comint-scroll-to-bottom-on-output
  all

I have included this little macro in my .emacs file with the following bit of elisp:

(fset 'autotest
   [?\C-x ?2 ?\C-x ?o ?\C-x ?3 ?\M-x ?s ?h ?e ?l ?l ?\C-m ?a ?u ?t ?o ?t ?e ?s ?t ?\C-m ?\M-x ?s ?e ?t ?- ?v ?a ?r ?i ?a ?b ?l ?e ?\C-m ?c ?o ?m ?i ?n ?t ?- ?s ?c ?r ?o ?l ?l ?- ?t ?o ?- ?b ?o ?t ?t ?o ?m ?- ?o ?n ?- ?o ?u ?t ?p ?u ?t ?\C-m ?a ?l ?l ?\C-m ?\C-x ?o ?\M-x ?e ?s ?h ?e ?l ?l ?\C-m ?\M-x ?s ?e ?t ?- ?v ?a ?r ?i ?a ?b ?l ?e ?\C-m ?c ?o ?m ?i ?n ?t ?- ?s ?c ?r ?o ?l ?l ?- ?t ?o ?- ?b ?o ?t ?t ?o ?m ?- ?o ?n ?- ?o ?u ?t ?p ?u ?t ?\C-m ?a ?l ?l ?\C-m])

Enjoy and let me know if you can think of any improvements. In particular, I would like to prompt for a directory path at the beginning and I need to figure out how to handle pre-existing *shell* or *eshell* buffers. Still, as is, this does what I need.

_PandoraJustRocks__

2006-04-05T12:48:00.000-07:00

For the first time ever, I find myself contemplating the use of a blink tag. Discretion has prevented it but it was a close race.

Pandora is incredible. My tastes in music have always been fairly eclectic. I like stuff from all over the board but I don't really understand why I like what I like. I have been using Pandora for a couple of days and I already understand more about my taste in music than 30+ years of developing that taste taught me.

As you listen to a custom play list based on Pandora's analysis of something you said you like (for example, I am listening to the Evanescence station right now.), take the time to explore you options. If you hear something you really like, give it a thumbs up and the qualities of that track will be used to select future tracks. If you hear something you don't like, give it a thumbs down. You will never hear that track again on that station. Also, Pandora seems to learn from that, and maybe the one difference that had from other tracks will be thrown out. In a frighteningly quick amount of time, artists I like start cross polinating channels and emerging spontaneously.

At any time if you want to know why Pandora suggested a track, you can ask it. It will give a quick summary of the musical properties that contributed to its selection. If you keep the process up, fairly quickly you will start to see common themes. If you pay attention to those descriptions, you start to see a pattern emerge. It turns out I really like minor key tonality, mixed acoustic and electric instrumentation, subtle use of vocal harmony and mild rhythmic syncopation. Woah. When I look back, almost everything that has ever knocked my socks off has combined a few of these features. Suddenly the most disparate of my tastes come together and make perfect sense.

I was totally unprepared to learn about myself from a random music link I found on the net. If you haven't tried it yet, why are you reading this tripe?! Expand your horizons and get yourself over to Pandora.

Opening Pandora's Box

2006-04-04T04:22:00.000-07:00

Wow. Just bloody freaking wow. I don't know how I got there but you have got to check out Pandora if you like music, any music, even just a little tiny bit. The interface isn't great. I have no idea where they got it but it really does suck. Still, if you can help me find more music in some of the obscure areas I like, then you are golden and Pandora does it with spades.

unit_diff is your friend

2006-04-03T19:04:00.000-07:00

Pat introduced me to unit_diff a while ago. At the time, it didn't resonate very well. There isn't much documentation and I didn't get what it was supposed to do. Now, I know exactly what it does and I can give first hand testimony to how great it is.

unit_diff is one of the tools included in ZenTest, along with zentest, multiruby and autotest. If you haven't installed it yet and you are using Test/Unit (You are using Test/Unit, right?), then you absolutely must grab the gem. $ sudo gem install ZenTest unit_diff runs diff on the expected and actual output of failed assert_equal tests in your test cases. Pipe your test output into unit_diff and relax with all the extra time you now have since you don't have to search through pages and pages of output to find the 2 characters that were different.

Actual command example I used recently:

$ ruby -Ilib test/test_parse_tree.rb | unit_diff
Loaded suite test/test_parse_tree
Started
.....FF.........................
Finished in 0.347 seconds.

1) Failure:
test_case_stmt2(TestParseTree) [(eval):1]:
6,10c6,8
<    [:case,
<     nil,
<     [:when, [:array, [:lit, 1]], [:lit, 2]],
<     [:when, [:array, [:lit, 3]], [:lit, 4]],
<     [:lit, 5]]]]]
---
>    [:when, [:array, [:lit, 1]], [:lit, 2]],
>    [:when, [:array, [:lit, 3]], [:lit, 4]],
>    [:lit, 5]]]]

2) Failure:
test_class(TestParseTree) [test/test_parse_tree.rb:370]:
Must return a lot of shit.
76,80c76,78
<      [:case,
<       nil,
<       [:when, [:array, [:lit, 1]], [:lit, 2]],
<       [:when, [:array, [:lit, 3]], [:lit, 4]],
<       [:lit, 5]]]]],
---
>      [:when, [:array, [:lit, 1]], [:lit, 2]],
>      [:when, [:array, [:lit, 3]], [:lit, 4]],
>      [:lit, 5]]]],

32 tests, 32 assertions, 2 failures, 0 errors

Without unit_diff, this was nearly 700 lines of output. Imagine how hard that was to process.

Debugging ParseTree (or Walking in Great Footsteps)

2006-04-03T18:12:00.000-07:00

In the course of creating CheckR, Pat Eyler and I discovered the following strangeness:

First example. Case with expression followed by multiple whens and an else.

$ echo 'case 1; when 2; 3; when 4; 5; else; 6;end' | parse_tree_show -f
[[:class,
 :Example,
 :Object,
 [:defn,
  :example,
  [:scope,
   [:block,
    [:args],
    [:case,
     [:lit, 1],
     [:when, [:array, [:lit, 2]], [:lit, 3]],
     [:when, [:array, [:lit, 4]], [:lit, 5]],
     [:lit, 6]]]]]]]
$

This results in a clear AST (abstract syntax tree).

Second example. Case with no expression followed by identical whens and else.

$ echo 'case; when 2; 3; when 4; 5; else; 6;end' | parse_tree_show -f
[[:class,
 :Example,
 :Object,
 [:defn,
  :example,
  [:scope, [:block, [:args], [:when, [:array, [:lit, 2]], [:lit, 3]]]]]]]
$

Since the only change was making case 1;' into 'case;, this AST is far from clear and is obviously missing things. There is no case node, no second when node and no else value. Unfortunately, for CheckR, we need the when nodes for our testing. Something has to be done.

Pat is on very good terms with drbrain (Eric Hodel) and zenspider (Ryan Davis) as the three of them ran Seattle.rb together. A few IMs later the bug was confirmed against the development branch of ParseTree. drbrain recorded a bug on the ParseTree RubyForge bug tracker and assigned it to zenspider.

Because I want to be a responsible netizen and (let's admit it) because I want to be cool like the big boys, I decided to create a test case for inclusion in the ParseTree test suite. Then, because it would be even cooler, and because Pat is always there egging me on, I thought I would take a whack at fixing ParseTree myself, with Pat's help.

Because of Pat's relationship with zenspider, we had access to a snapshot of the development tips of ParseTree. zenspider is not using RubyForge to host the dev code, so not everyone has access to this. As a result, I will document the rest of this story as if we had access to the same things the world has access to. All references to source code will be to the released code for ParseTree-1.3.7.

First the test case. This is very likely to be wrong since I don't have a correct AST of the case;when syntax but it is a start. ParseTree uses two files to dynamically generate its test case, something.rb and test_parse_tree.rb. something.rb contains method and class definitions to be parsed. test_parse_tree.rb contains expected ASTs to be compared against the ASTs generated during the tests. All it takes to add a new test case is add the method definition to something.rb and the expected AST to test_parse_tree.rb.

Before we go mucking things up, we need to know where the test suite stands as released. We have to make sure we don't break anything else when we add our own tests.

The following results were produced on Cygwin. It was also necessary on Cygwin to comment out $TESTING = true in test_composite_sexp_processor.rb and test_sexp_processor.rb in order to get RubyInline to work correctly when compiling for the test. Many thanks to drbrain for his patience and help with that particular problem.

$ cd /usr/lib/ruby/gems/1.8/gems/ParseTree-1.3.7
$ make test
Loaded suite test/test_all
Started
.....F.F..F......F..............................F................................
Finished in 0.328 seconds.

-- output snipped due to length --

81 tests, 94 assertions, 5 failures, 0 errors
make: *** [test] Error 1
$

If I recall, there were only 4 errors in linux when I ran this same test. Either way, it is very important to know about the failing tests before we start. We don't want to assume we broke more than we intended when we add our tests. (As an asside, the dev branch doesn't have these errors, allowing us to be even more confident in what we do.) Now, on to the test.

in test/something.rb

def case_stmt2
 case
 when 1
   2
 when 3
   4
 else
   5
 end
end

This is fairly simple but contains everything I know about the problem to date. Namely, ParseTree seems to eat the case, when 3 and else.

in test/test_parse_tree.rb

 @@case_stmt2 = [:defn, :case_stmt2,
   [:scope,
     [:block,
       [:args],
       [:case,
         [:when, [:array, [:lit, 1]], [:lit, 2]],
         [:when, [:array, [:lit, 3]], [:lit, 4]],
         [:lit, 5]]]]]

At first glance, it seems fairly difficult to generate this and it would have been if I had done it by hand. The easy way was to run the following command, steal its output and munge it just a little bit.

$ echo 'class Blah; def case_stmt2; case true; when 1; 2; when 3; 4; else; 5; end; end; end' | parse_tree_show
[[:class,
 :Blah,
 :Object,
 [:defn,
  :case_stmt2,
  [:scope,
   [:block,
    [:args],
    [:case,
     [:true],
     [:when, [:array, [:lit, 1]], [:lit, 2]],
     [:when, [:array, [:lit, 3]], [:lit, 4]],
     [:lit, 5]]]]]]]

Looking at the other expected trees, we see they all start with the method definition. Also, we need to remove the [:true] because it doesn't appear in our test method. I want to reiterate, the resulting expectation, recorded above in test_parse_tree.rb may not be correct but it is a place to start.

Finally, we run our test again.

$ make test
ruby  -w -Ilib:bin:../../RubyInline/dev test/test_all.rb
Loaded suite test/test_all
Started
.....F.F..FF......F..............................F................................
Finished in 0.354 seconds.

-- output snipped due to length --

  3) Failure:
test_case_stmt2(TestParseTree) [(eval):1]:
<[:defn,
 :case_stmt2,
 [:scope,
  [:block,
   [:args],
   [:case,
    [:when, [:array, [:lit, 1]], [:lit, 2]],
    [:when, [:array, [:lit, 3]], [:lit, 4]],
    [:lit, 5]]]]]> expected but was
<[:defn,
 :case_stmt2,
 [:scope, [:block, [:args], [:when, [:array, [:lit, 1]], [:lit, 2]]]]]>.

-- output snipped due to length --

82 tests, 95 assertions, 6 failures, 0 errors
make: *** [test] Error 1
$

One new test failure and it's our test. Perfect. Also, if you look closely, you will see our new test is included in the massive output of test_class. That's important because even if we started with no test failures at all, our addition of one test would cause two failures. test_class is an integration test of the class as a whole (including all methods) and ensures that working methods still parse in a sane way as a class and also that the AST for the class definition itself is correct.

Now, any sane person would stop here, submit the test to the ParseTree developers and call it a day. Unfortunately, neither Pat nor I appears to be sane, so we started chasing the cause of the failure.

To Be Continued....

Convictions

2006-03-15T06:42:00.000-08:00

Recently, I am becoming convinced of a couple of things. First, emacs is good. This one is growing on me. I still suck completely at emacs but the more I learn, the more I think there is to like. Second, I have lost it. I am not a great programmer. I am pretty sure I used to be but I have descended from great to merely good. Something is missing. I have lost the drive that used to make me devour all things programming. I wonder if I can get it back.