How has Ruby blown my mind?

In a word iterators. No single language feature has changed the way I think about code more than the simple, now seemingly obvious, pervasive, compelling act of passing anonymous code into a method to be run in context by that object. In a few days, this feature became so natural to me that languages where it was missing - Java, C, Perl - were harder just because I knew it was missing.

By the way, if you don't know about Pat Eyler's & Apress' ongoing blog about Ruby contests, check out this month's contest.

Install MIT Scheme

Pat has convinced me to set some minor goals on the way to completing Structure and Interpretation of Computer Programs together. The first goal is to get the development environment installed. We have agreed to complete it by Monday, September 11th.

See more progress on: Read SICP

ActiveRecord, Delegation and Demeter

Many of you have heard of the Law of Demeter (LoD). For those who haven't, I recommend reading Robert C. Martin's great article (in his Engineering Notebooks), the Dependency Inversion Principle.

There are many great descriptions of LoD online but in practice, it boils down to dot chains are usually bad. (E.G. foo.bar.baz) The reason they are bad is because they generally represent a place where encapsulation is broken. In my example, the currently running code knows that foo provides bar, which is ok, but it also knows that bar provides baz. Really, the currently running code should only know the methods on foo. If it needs baz, it should call foo.baz. Note that this is not the case with Java's sb = new StringBuffer(); sb.append(str).append(str); since in this example, you are working on sb each time and not descending down into sb's children.

The problem with trying to implement LoD in real code is that short of a complicated framework for avoiding the dot chains, you tend to end up writing lots of small delegation methods and you tend to have to write new ones every time a child's implementation changes. Obviously this becomes fragile over time and explains why there are refactorings for adding and removing delegation in Martin Fowler's famous book, Refactoring.

Fortunately, I don't code in average languages anymore. These days, I am all about Ruby. In particular, I am usually writing Ruby in the context of Rails. Over the last couple of months, my boss and I have developed an elegant way to add delegation to ActiveRecord objects. Our technique could be extrapolated to a more general case but ActiveRecord models is where we want it so that is all we have done.

The following example explains how our code used to work.

class User < ActiveRecord::Base; end
class Agent < User
  has_one :agent_detail
end
class AgentDetail < ActiveRecord::Base
  belongs_to :agent
  belongs_to :address
end
class Address < ActiveRecord::Base
  has_one :agent_detail
end

This is backed by a database with these tables:

users: id, type, *
agent_details: id, agent_id, address_id, *
address: id, *

So, agents are users (and exist in the users table) but since agent needs more fields than user does and we didn't want to pollute the users table with the needs of every kind of user, we provide an agent_detail table that contains all the information that agents need and users don't. Also, since addresses are a cross-cutting concern in an application, we decided to store all addresses in one table which can be referenced by others like agent_details.

So, say you have an agent bob. To get bob's zip code, you would have to do bob.agent_detail.address.zip_code, breaking encapsulation and delving far too deep into implementation details of an agent. What we really want to be able to do is bob.zip_code. Our code shouldn't care how bob implements this field but it should be able to get this information directly from bob. We also need to be able to set this information on bob with bob.zip_code = '12345'. The normal and naive way to do this follows:

class Agent < User
  has_one :agent_detail

  def zip_code
    agent_detail.zip_code
  end

  def zip_code=(zip_code)
    agent_detail.zip_code = zip_code
  end
end
class AgentDetail < ActiveRecord::Base
  belongs_to :agent
  belongs_to :address

  def zip_code
    address.zip_code
  end

  def zip_code=(zip_code)
    address.zip_code = zip_code
  end
end

This allows us to do bob.zip_code but it has some problems. For one, at the moment, nothing is ensuring all agents have an agent_detail and all agent_details have an address. This means we are subject to nil problems (also know as NullPointerExceptions in the Java world). Also, say we wanted to change the name of zip_code to zip? We would have to change a table and a total of 8 lines of code in 2 different classes. Not exactly the most robust system. This also has some problems with saving dependent objects. If you do bob.zip = '12345'; bob.save, odds are the you expect the zip code change on address to be saved but it isn't.

Ruby and Rails have some great tools that can help us simplify the code above and remove some of the fragility of the interface in the face of change. Specifically, method_missing comes to the rescue to handle the bob.zip_code method and delegate it down and before_save and after_safe ActiveRecord callbacks help solve the save problems. This makes your code look like this:

class Agent < User
  has_one :agent_detail

  after_save {|agent| agent.agent_detail.save}

  def safe_agent_detail
    agent_detail.nil? : build_agent_detail ? agent_detail
  end

  def method_missing(method_id, *args)
    if safe_agent_detail.respond_to? method_id
      agent_detail.send(method_id, *args)
    else
      super
    end
  end

  def respond_to?(method_id)
    safe_agent_detail.respond_to?(method_id) || super
  end
end
class AgentDetail < ActiveRecord::Base
  belongs_to :agent
  belongs_to :address

  before_save {|agent_detail| agent_detail.address.save}

  def safe_address
    address.nil? : build_address ? address
  end

  def method_missing(method_id, *args)
    if safe_address.respond_to? method_id
      address.send(method_id, *args)
    else
      super
    end
  end

  def respond_to?(method_id)
    safe_address.respond_to?(method_id) || super
  end
end

This does neatly solve most of our problems but there is still one nagging itch, this code isn't very DRY. Most of the details in AgentDetail and Address are the same. What if we could take a page out of Rails and do this:

class User < ActiveRecord::Base; end
class Agent < User
  has_one_delegate :agent_detail
end
class AgentDetail < ActiveRecord::Base
  belongs_to :agent
  belongs_to_delegate :address
end
class Address < ActiveRecord::Base
  has_one :agent_detail
end

This is in fact, what my boss and I have done. The following code provides has_one_delegate and belongs_to_delegate:

module ActiveRecord
  class Base
    def self.belongs_to_delegate(delegate_id, options = {})
      delegate_to(:belongs_to, delegate_id, options)
    end

    def self.has_one_delegate(delegate_id, options = {})
      delegate_to(:has_one, delegate_id, options)
    end

    private

    def self.delegate_to(macro, delegate_id, options = {})
      send macro, delegate_id, options

      save_callback = {:belongs_to => :before_save, :has_one => :after_save}[macro]

      send save_callback do |model|
        model.send(delegate_id).save
      end

      delegate_names = respond_to?('safe_delegate_names') ? safe_delegate_names : []
      delegate_names = (delegate_names - [delegate_id]) + [delegate_id]

      def_string, source_file, source_line = <<-"end_eval", __FILE__, __LINE__

        def self.safe_delegate_names
          #{delegate_names.inspect}
        end

        def safe_delegates
          self.class.safe_delegate_names.collect do |delegate_name|
            send(delegate_name).nil? ? send("build_\#{delegate_name}") : send(delegate_name)
          end
        end

        def method_missing(method_id, *args)
          safe_delegates.each do |delegate|
            return delegate.send(method_id, *args) if delegate.respond_to?(method_id)
          end
          super
        end

        def respond_to?(*args)
          safe_delegates.any? {|delegate| delegate.respond_to?(*args)} || super
        end

      end_eval

      module_eval def_string, source_file, source_line + 1
    end
  end
end

The code itself isn't the most beautiful internally, but the simplicity it provides is elegant and effective. Also, because we eventually did need more than one delegate for the same class, it is robust enough to allow multiple delegates. Score one for the Ruby/Rails team. An entire class of refactorings goes from recipe to method call. This is why I love programming.

Eventually, we plan to release this as a plugin to the Rails community. Until then, let me know what you think. Would you use it?

When Inheritance Doesn't Pay - Problems Testing Rails

The smallest things have the capacity to cause the most pain. Today's pain is caused by 5 characters, or to be more acurate, the absence of 5 characters.

super

Say it with me, super. Yep, super. Guess what Rails doesn't include in the setup of functional tests by default. You betcha, super. And the funny thing is, most of the time, it really doesn't matter. But sometimes, oh boy, yeah, sometimes it makes all the difference.

Given two test classes:

a_test.rb
require File.dirname(__FILE__) + '/../test_helper'

class ATest < Test::Unit::TestCase
fixtures :empties

def setup
end

def test_fixture
empties(:first)
end
end

b_test.rb
require File.dirname(__FILE__) + '/a_test'
class BTest < ATest; end

Would it surprise you that run seperately, both of these pass with flying colors, but run together, they result in abominable crashes? It sure surprised me.

[~/projects/test]$ /usr/bin/ruby -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" "test/unit/a_test.rb" Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
Started
.
Finished in 0.115591 seconds.

1 tests, 0 assertions, 0 failures, 0 errors
[~/projects/test]$ /usr/bin/ruby -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" "test/unit/b_test.rb" Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
Started
..
Finished in 0.157902 seconds.

2 tests, 0 assertions, 0 failures, 0 errors
[~/projects/test]$ /usr/bin/ruby -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" "test/unit/a_test.rb" "test/unit/b_test.rb"
Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
Started
EEEE
Finished in 0.088633 seconds.

1) Error:
test_fixture(ATest):
NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occured while evaluating nil.[]
/usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/fixtures.rb:475:in `empties'
./test/unit/a_test.rb:10:in `test_fixture'

2) Error:
test_fixture(ATest):
NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occured while evaluating nil.-
/usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/transactions.rb:112:in `unlock_mutex'
/usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/fixtures.rb:534:in `teardown'

3) Error:
test_fixture(BTest):
NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occured while evaluating nil.[]
/usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/fixtures.rb:475:in `empties'
./test/unit/a_test.rb:10:in `test_fixture'

4) Error:
test_fixture(BTest):
NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occured while evaluating nil.-
/usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/transactions.rb:112:in `unlock_mutex'
/usr/lib/ruby/gems/1.8/gems/activerecord-1.14.2/lib/active_record/fixtures.rb:534:in `teardown'

2 tests, 0 assertions, 0 failures, 4 errors
[~/projects/test]$

The key to this dilemma should be pretty easy to find now that I have bloodied myself on the problem for you. Exactly, super. Add super to the setup and all our woes are cured. I haven't got the slightest idea what magic TestCase.setup is doing but without it, we bomb every time. Actually, I suspect inheritance isn't even the culprit here. Running a with itself twice exhibits the same behavior, with the same solution.

I suspect the error condition is related to the edge case of a test class completing then being loaded again but I'm not sure how to prove it. I'm guessing the 2nd time through, an action isn't take because Rails is convinced it already happened. But the moral is clear, super is your friend.

Compounding Errors - Another Rails Testing Adventure

The Moral:
Always include test_helper in Rails tests.
Be specific when handling exceptions.
Use diff. Trust diff.
Pair often. Pair always.

The Story:
Every test class generated by Rails includes require File.dirname(__FILE__) + '/../test_helper'. If you remove it, or you write your own test and forget it, bad things can happen, things like this....

First things first, how are the tests? rake test_functional. Failure. Hmm. I expected that. This project was written before we fully committed to test first. It shouldn't be too hard to get the existing tests to run. Let's see what we need.

The failure is in the functional tests. A require is failing and I don't know why. I wonder where that file is supposed to come from? Maybe it is really missing. There is something similar in another one of our projects. For now, I will just include the mock we used to make that test pass. (Copy the test/mock directory from one rails app to the other.)

That should fix it. rake test_functional. Blamo. Still failing. Hmm. Naming problems? File names seem to match. Hack. Hack. Hack. This sucks, I've burned an afternoon and there's nothing to show for it.

Interlude. A night of little sleep and a nagging weight on my mind. Not fun. The tension wakes me up at 3:00 AM and I start hacking from home. It isn't like I was getting any useful rest anyway.

So, the test isn't finding my mock either. I don't get it. Well, I guess I can check my load path before the require.

puts $:.inspect
 ["lib", "test", "/usr/lib/site_ruby/1.8", "/usr/lib/site_ruby/1.8/i386-linux",
 "/usr/lib/site_ruby", "/usr/lib/ruby/1.8", "/usr/lib/ruby/1.8/i386-linux",
 "."]

That doesn't look right. None of my rails includes are there. Hack. Hack. Hack.
Burned several morning hours and I still have to face Steve and tell him I have nothing to show for it. Having that hanging over my head sure does make for a long morning commute.

OK. Before I talk to Steve about this, lets at least check his machine. rake test_functional Passed. Everything passed. Wait. WTF do you mean everything passed?! OK. Maybe I'm just an idiot (I mean past experience suggest this must frequently be the case.) so lets make sure I have the current code. svn status. Nothing modified. svn udpate. Nothing missing. Well, maybe Steve made changes on his machine. svn status. Nothing modified. svn udpate. Nothing missing.

I've been here before. Code works on one machine but not on the other. The problem is, this time, it is well tested code and I can't find anything that is different. Discussing it with Steve didn't help either. He spent a chunk of time getting all these tests to pass just recently so they should still work.

Well, my machine is different. We are planning a system upgrade and I'm the guinea pig. But most of our software is the same. Sure, the OS is different right now but.... Well, I can at least standardize my software install. I'll just get our environment setup script to work.

Great. That script blows up on my machine too. It can't get the file from subversion. Well, of course it can't. That doesn't look like the repository location to me. Hack, hack. Oh. Idiot. That was right. The subversion repository is being served up by http and I was looking for the svn+ssh location.

Well, it still blows up. Compare the file path. Still missing. Load the directory in a browser. There the file is. But, umm, well. OK. Maybe it is something in the version. That looks different in the script. Nope. That looks right.

Eventually, I notice the file name the script is looking for is .tar.bz2 but the file name in the directory is .tar.gz. Don't ask how long it took. Too long.

It's looking for the wrong file name. Why. Where does that come from. There it is. But, wait, it should be looking for the name that is there first. If it doesn't find that then it catches the exception and tries the other name.

The light bulb finally goes on.

Ah. I'm getting an exception other than the expected exception and that is incorrectly being understood as a missing file. And hey, while I'm at it, guess what, I should be running this as root and I'm not. The real error is a permissions error but I can't see that because we handled every exception instead of just the missing file exception we expected. And surprise, surprise, everything works great when I run it as root. I leave the script to run over night since it is taking too long.

Interlude. Another bad night. Another early morning. This time I was so burned out that instead of going to hacking night, I crashed for 3 hours when I got home. Of course that meant I couldn't get back to sleep until the wee hours. And of course that didn't stop me from waking up way too early.

Back at the office again. Everything is good on Steve's machine, and not good on my machine. Well, lets diff the output. I should have done this days ago but hindsight is 20/20 and foresight is blind as a bat. Well, the commands for the tests aren't in the same order but I don't really see anything else. Lets follow the rake trace.

The trace is different but I don't really understand why. Hack. Hack. Oh, by the way, things work in autotest but not rake. Now that is just creepy.

Finally, I corner Steve to help me out. I'm not solving this alone, so lets try some of that pairing juju.

Hack here. Hack there. Trace this. Trace that. Eventually Steve notices what I already knew but this time, Steve gets it when I just blanked. It is order dependent. The tests are being run in a different order. Hack, hack. Well, we can't really control the order, it seems to be an artifact of the different OS. Still, we should be able to figure out what causes the order dependency. Hack. Hack.

Bingo, there it is. This test (which was copied from another, non-rails project) doesn't include test_helper. And guess what. test_helper is where environment.rb is loaded. And environment.rb is where the additional include paths are added, not to mention that test_helper is where the mock paths are added. No wonder it couldn't find those mocks that I added.

Finally, we get to here and now. Two and a half days of developer time wasted over several small errors, compounded together. Cory Foy recently wrote "And that's the trick with being Agile. If you slip, even for a second, it *will* bite you. You have to stay on guard constantly". This is true of Agile but really, it is true of all development. How many times have you been bitten by small problems that take days to figure out and would have been avoided with just that little bit more effort?

Of course, the mark of a great developer isn't the lack of mistakes. The mark of a great developer is learning from mistakes and preventing them in the future. Soon, we will either have something that tests to make sure our tests include test_helper, or even better a way to avoid that step in every single bloody test file. We will also be more careful to trap only what we expect and leave the unexpected unhandled or at least properly handled. And I will definitely learn to trust my diffs earlier.

Don't make my mistakes, but more importantly, don't make your mistakes twice.

Assignment in Ruby - Block Assignment

For the second installment of Assignment in Ruby, I would like to explore block assignment. Block assignment is very similar to the simple scoped assignments I explored in my first post. To understand the differences, it is necessary to understand how block variables are scoped.

When a block uses a local variable, Ruby has to determine what is being used. { |x, y, z| puts x, y, z } uses three local variables. Interestingly, these three variables could each potentially have a different scope.

If there is an x in the current local scope, the variable x used in the block will be that local variable, meaning the assignment will be an :lasgn (local assignment).

If there is no y in the current local scope but there is a y in an enclosing block, the y variable used will be the block scoped variable from the outer block and the assignment will be a :dasgn (block assignment).

If there is no z in the current local scope and there is no z in any enclosing blocks, then a new z will be created in the scope of the current block and the assignment will be a :dasgn_curr (block local assignment).

The different scopes are each illustrated in the following code:

$ echo "x = 123; foo() {|x| y = 456; bar() {|x,y,z| puts x, y, z}}" | parse_tree_show -f
[[:lasgn, :x, [:lit, 123]],
 [:iter,
  [:fcall, :foo],
  [:lasgn, :x],
  [:block,
   [:dasgn_curr, :y],
   [:dasgn_curr, :y, [:lit, 456]],
   [:iter,
    [:fcall, :bar],
    [:masgn, [:array, [:lasgn, :x], [:dasgn, :y], [:dasgn_curr, :z]]],
    [:fcall, :puts, [:array, [:lvar, :x], [:dvar, :y], [:dvar, :z]]]]]]]

Written a little more nicely, the code under inspection is:

x = 123
foo() { |x|
  y = 456
  bar() { |x,y,z|
    puts x, y, z
  }
}

Note the 2nd to last line of the S expression [:masgn, [:array, [:lasgn, :x], [:dasgn, :y], [:dasgn_curr, :z]]]. The first thing to notice is:masgn. :masgn stands for multiple assignment and will be covered in its own post. After the :masgn you can see concrete examples of the local assignment to x, the outer block assignment to y and the current block assignment to z.

This can be a sticky situation and I know I missed it when I was learning Ruby. I only internalized how block scopes are handled when I saw it reflected in ParseTree and the AST. I still don't really understand why you would want to reuse local scoped variables inside a block but that is probably an indication of my weakness with closures. If you have any ideas, please let me know.

I hope you have enjoyed another brief forray into Assignment in Ruby. Join me next time as we explore multiple assignment and how it can simplify your life.

Climbing without a Net

Developing test first is like rock climbing on lead.

When a climber climbs lead, they have to place protection as they go. At any point, if they fall, their last piece of protection generally keeps them from falling far. Sometimes, that protection will fail and when it does, the other protection they have placed keeps them from death or serious injury.

Each test written is like a piece of protection we have placed. Usually, the last test we wrote is the most important but all the tests together keep us from serious injury. Our tests really are our safety net and I can't understand why people choose to live without it.

If test first is lead climbing, then it goes to reason that the test first challenge is top roping and developing without tests is free climbing. Sure the best climbers can skate up easy climbs without protection but for the rest of us, that is the path to certain doom.

When Rails Needs a Clue - Single Table Inheritance Problems

Given Grandchild < Child < Parent < ActiveRecord::Base, your controllers need model :grandchild. Without it, well, this could happen:

There is a sinking feeling in your gut. You've been working in rails for weeks. You've been doing everything right, Test Driven Development to the hilt, when suddenly, the worst happens. Behavior in test doesn't match behavior in development. You begin to question whether any of your testing is valid. Does your app really work, or does it just seem to work in test? How can test and development possibly differ? Your worst nightmares are coming true.

This might seem like a ridiculous scenario but it is exactly what I have dealt with for the last week. We have a table with single table inheritance. Admin < Broker < User < ActiveRecord::Base all in table :users. Errors started showing up in development that passed in test. We tracked it down to differing sql statements. In test, Broker.find_all generated SELECT * FROM users WHERE ( (users.`type` = 'Broker' OR users.`type` = 'Admin' ) ). In development, the exact same Broker.find_all generated SELECT * FROM users WHERE ( (users.`type` = 'Broker' ) ).

We had no idea what was going on and we had features that had to be ready right now for a demo. We worked around the problem. We wrote our own find conditions and moved on, writing a todo to find out what was wrong. The sleepless nights started and the worry began to grow.

Over the weekend, the other developer on my team looked at some code I wrote Friday and had problems. He didn't have time to look into it but told me about it this morning. My first response, well, I'll write a test that shows the problem, only every test I wrote worked fine in test. I could reproduce his problems in development every time. This shouldn't be so hard.

Start from the basics. I created a test with exactly the same input as was reported in the error on development. Same parameters. Same session. Same flash. Everything was exactly the same. Only, what failed in development passed in test. The sick feeling in my stomach is getting stronger. I feel like I'm in a dream watching my career go down the drain. Sure enough, I track it down to nearly the same problem as before. We have Default < ActiveRecord::Base; belongs_to Broker. If the default.broker_id pointed to a user with type=Admin, it failed in development. Default.broker was returning nil when it should have been returning an Admin. Again, the SQL was different between test and development in exactly the same way. Development would only accept Brokers and test accepted Brokers and Admins. This didn't make any sense and this time, we had time to look at it.

My partner started looking at the source in Rails and I started searching online. I didn't find anything with a few quick searches. I knew the #rubyonrails channel on Freenode is pretty active so I decided to ask my question there. "Has anyone seen differences between test and development when..." Hmm. I want the exact SQL I'm seeing in test and development before I ask this question. I know, I'll fire up the console for both and get it from the logs that way.

Loki took pity on me here and Coyote found something else to toy with. export RAILS_ENV=test; ./script/console. So far, so good. Broker.find_all; I only get Brokers back. WTF! That's the same thing I've been seeing in development. A quick check. Yes, I'm in test. Hmm. User.find_all; I get everything. (select * from users, no where clause) Admin.find_all; I get only Admins. I expected that, Admin doesn't have any children. Broker.find_all; Holy hand grenades. That is everything, Admins included. The where clause changed. Umm. But. Maybe this time I put something different. Just to be sure, I used up arrow to run the first Broker.find_all; (I couldn't see a difference but who knows at this point.) Sure enough, the original search turns up the new results. Then, wham, lightning struck. If Admin has never been loaded, Rails doesn't know Admin < Broker!

A quick conversation with my partner, a little test (added model :admin to the controller) and sure enough, everything is peachy. There is just one nagging doubt. I don't know how to create this in test. You see, single table inheritance is implemented in a single YAML fixture. When fixtures :users runs, Rails figures out I have a User, Broker, Customer (< User) and Admin model. Loading my fixture seems to make Rails aware of those relationships. So, how do I get data for admins into the table for users without making Rails aware of the Admin class? I have no idea.

The sinking feeling isn't gone but it is a good deal smaller. I now have a picture of the world that explains the unexplainable and test and development work the same again. But still, that little doubt.

If you know how to test this, won't you please let me in on the secret?