~/scratch

Schema loading for tests in rails

2011-02-04T04:21:00Z

I’m reading The Rails 3 Way by Obie Fernandez, and it says

Every time you run tests, Rails dumps the schema of your development database and copies it to the test database using an autogenerated schema.rb script.

In my experience this isn’t quite what happens. From what I’ve observed:

schema.rb gets generated when you call rake db:migrate
the schema gets loaded from schema.rb to your test database when you call rake db:test:prepare
when you run tests, the calls to the database inside each of the tests are run within a transaction that gets rolled back at the end of the test

In practice, you could therefore load seed data into your test database after calling rake db:test:prepare and you wouldn’t lose this every time you run your tests.

namespace :db do
  namespace :test do
    desc "Init bare bones test data"
    task :seed_db do
      # your seed data here
    end
  end
end

Rake::Task["db:test:prepare"].enhance do
  Rake::Task["db:test:seed_db"].invoke
end

Is this a good idea? Well, It depends. I’m working on one project where it is extremely helpful. Perhaps when we’ve gotten things refactored a bit it won’t be necessary anymore.

Rails 3 Scopes: With great power comes great responsibility.

2011-01-29T08:39:13Z

I wrote one of the worst bugs of my career (so far) the other day.

The long and short of it is this:
ActiveRecord::Relation may behave like an array in many ways, but it’s not an array. If you are doing something which is completely harmless to an array, but could be potentially destructive to an active record relation, make sure what you have really is an array.

It goes like this

Create a fresh rails3 project, and set it up to use rspec.

$ rails new todo -T
# Gemfile
source 'http://rubygems.org'
gem 'rails', '~> 3.0'
gem 'sqlite3-ruby', :require => 'sqlite3'
gem 'rspec-rails', '~> 2.4'

rails g rspec:install

Create a simple model.

rails g migration create_tasks

class CreateTasks < ActiveRecord::Migration
  def self.up
    create_table :tasks do |t|
      t.column :description, :text
      t.column :done, :boolean, :default => false
      t.timestamps
    end
  end

 def self.down
    drop_table :tasks
  end
end

rake db:migrate && rake db:test:prepare

Add a scope

class Task < ActiveRecord::Base
  scope :done, where('done = ?', true)
end

This is where it gets interesting.

require 'spec_helper'

describe Task do
  it "cleanly extracts from array" do
    trucs = {:des => :trucs}
    machin = {:un => :machin}
    bidule = {:une => :bidule}

    stuff = [trucs, machin, bidule]

    extracted = stuff.delete(machin)
    extracted.should == machin
    stuff.should_not include(machin)
  end

  context "active record relation" do
    it "extracts cleanly" do
      jump = Task.create(:description => 'jump')
      run = Task.create(:description => 'run')
      walk = Task.create(:description => 'walk')

      activities = Task.all # activities.class == Array

      extracted = activities.delete(run)
      extracted.should == run
      activities.should_not include(run)

      Task.all.should include(run)
    end

    it "doesn't. Sneaky bastard" do
      jump = Task.create(:description => 'jump', :done => true)
      run = Task.create(:description => 'run', :done => true)
      walk = Task.create(:description => 'walk', :done => true)

      activities = Task.done # activities.class == ActiveRecord::Relation

      extracted = activities.delete(run)

      Task.all.should include(run)

      # fails
      # incidentally, also fails on these:
      # popped.should == run
      # activities.should_not include(run)
     end
  end
end

To quote the log:

AREL (0.2ms) DELETE FROM “tasks” WHERE (done = ‘t’) AND (“tasks”.“id” = 2)

Yeah. Potentially really not good.

PostgreSQL template tables and rake db:test:prepare

2011-01-13T04:00:00Z

I recently overcame the final hurdle to getting rspec hooked up within this legacy project that I’m working on (legacy in the sense that it has no tests, and is in production, and works).

There were three levels of fail with respect to running rake db:test:prepare

Fail #1: Configuration

The database configuration file had the test environment looking at the same database as development. The first time I ran rake db:test:prepare it dropped my development database. Usually this isn’t such a huge disaster, but in this case we’re using a slightly sanitized version of the production database. It takes 45 minutes to scp it down from the backup server, and 4+ hours to load.

The solution, of course, was to change this:

test:
  adapter: postgresql
  database: legacyproject_development

to this:

test:
  adapter: postgresql
  database: legacyproject_test

Fail #2: Bug in Schema Dumper

This project happens to be using a homegrown plugin for PostGIS (because at the time we needed one there were no other available options). Since we’ve never needed to actually load the database from the schema, nobody had noticed that the schema definition of geometry tables were lacking the parameter for the SRID.

wrong number of arguments (4 for 5)

Fixing the custom schema dumper for the geometry tables fixed this issue.

Fail #3: Missing PostGIS Functions and Tables

The production and development databases had been set up manually to include the PostGIS goodies. rake db:test:prepare was now correctly creating the test database based on the schema, but was lacking everything PostGIS. Three options immediately came to mind:

write a task that manually loaded all the postGIS stuff and have it run prior to the rake db:test:prepare task (uhm, no thanks).
add the PostGIS stuff to the default postgresql template table, template1 (I’d rather not).
create a special PostGIS template table (yes, please).

So I did:

psql -d postgres -U postgres
CREATE DATABASE template_postgis WITH TEMPLATE=template1 ENCODING='UTF8';
\c template_postgis;
CREATE LANGUAGE plpgsql;
\i /opt/local/share/postgresql90/contrib/postgis-1.5/postgis.sql
\i /opt/local/share/postgresql90/contrib/postgis-1.5/spatial_ref_sys.sql
UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template_postgis';
GRANT ALL ON geometry_columns TO PUBLIC;
GRANT ALL ON spatial_ref_sys TO PUBLIC;

And then all I needed to do define the database config so that it used this template instead of template1 when created the test database.

test:
  adapter: postgresql
  template: template_postgis
  database: legacyproject_test

Ah, right. Not so simple. It would seem that noone has needed this config options, so rails doesn’t recognize the template option.

So we submitted a patch to rails. Smallest patch in the history of rails probably, at less than 30 characters

#win

A (silly) PostgreSQL gotcha

2011-01-07T09:04:09Z

As I was setting up my latest project in Rails 3 with PostgreSQL, Cucumber, RSpec and various other goodies, I got this error when running rake db:migrate:

PGError: ERROR: permission denied for relation schema_migrations
SELECT version FROM schema_migrations

I took a quick look in my config/database.yml

And saw the following ridiculousness:

development:
  adapter: postgresql
  encoding: unicode
  database: myproject_development
  host: localhost
  pool: 5
  username:
  password: productionpassword

Updating username to myproject and password to be blank did the trick.

Using PostgreSQL with Rails 3, Cucumber & RSpec

2011-01-05T08:55:54Z

One thing that tripped me up when trying to set up a new project using PostgreSQL rather than sqlite3, is that when running rake db:test:prepare I got the error message:

/path/to/rake:19:in `<main>'
Couldn't create database for
{"adapter"=>"postgresql", "encoding"=>"unicode",
"database"=>"myproject_test", "host"=>"localhost",
"pool"=>5, "username"=>"moi", "password"=>nil, "min_messages"=>"WARNING"}
rake aborted!
FATAL:  database "test" does not exist

This despite having just run createdb -O moi test

I recreated the database, checked that it was, in fact, there (psql myproject_test, and ran the rake command again. Boom. Same error message.

Turns out the user “moi” was allowed to drop databases, but not create them.

psql myproject_development;
ALTER USER moi CREATEDB;
\q

Et voila.

Lazybones: Skeleton Rails3 App

2010-12-28T07:54:00Z

I’ve set myself a 30 day challenge for the month of January. Nevermind that January has 31 days.

Every morning, I’m going to get up an hour early (yes, weekends included) and practice programming.

This is different from programming at work. For one, I don’t need to deliver anything. I basically want to be able to do some deliberate practice on TDD/BDD and refactoring, as well as explore some gems that I’m unfamiliar with.

I have a couple of projects that I can use for this. One is OverkillCMS:
git://github.com/kowen/overkill.git

This is a simple CMS for a friend of mine which I’m making simply to practice full TDD/BDD using rspec and cucumber. That, and because her current website requires her to edit html and css, and she’s not a programmer.

Another is Carbon Dating – An Open-Source Ruby on Rails Dating App
git://github.com/kowen/c14.git

This one is also inspired by some friends of mine. They’re running a tiny niche dating site that uses some of the ugliest PHP code you will ever see. It’s an embarrassment to cowboy coders everywhere.

In preparation for these two projects, I went ahead and found out how to set up a rails 3 project per my own preferences (rspec, cucumber, jasmine, haml, sass, compass, watchr, metric_fu) and documented it here, along with some convenience files:

git://github.com/kowen/lazybones.git

Zeo: Hacking Sleep

2010-12-18T11:32:03Z

I purchased a zeo.

I received it yesterday, have tried it once, and so far I’m impressed. It’ll be exciting to see how much data I can get out of it.

It was able to report:

how long it took me to fall asleep (33 minutes)
how long I slept, total (10 hours, 25 minutes)
how many times I woke up during the night (6. Ouch. I only remember 3 of them.)
how much deep, light, and REM sleep I got (plenty!)

The headband was comfortable and non-disruptive. I toss a bit, and slept on my sides and my stomach, and the headband didn’t budge.

My ZQ score was 114. Sky high, according to the pamphlet they included in the package. Apparently in my age bracket the average is around 80.

Interestingly, when I woke up, my subjective judgement was that I had slept “okay”.

Encoding issues: cucumber

2010-12-17T20:27:00Z

In my standalone test runner which uses ruby 1.9.2, cucumber driving firewatir, and using rspec matchers I have a test that looks like this:

Scenario: Single logout flash
   Given I am logged in with "username"/"password"
   When I log out
   Then I should see "Du er nå logget ut"

That last bit is in Norwegian. The project this code tests has support for two Norwegian written languages, and English (sort of). Actually, let’s not go there.

The browser is set up like this:

Watir::Browser.default = "firefox"

def browser
  @browser ||= Watir::Browser.new
end

The test should be passing, but I get the following error on the line with the Norwegian text in it.

incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)

Tthe step definition for the failing step is:

Then /^I should see "([^"]*)"$/ do |term|
  browser.text.should include(term)
end

ASCII-8BIT isn’t really an encoding, as far as I can tell. It’s unencoded bytes (and is therefore aliased to BINARY).

There are a few places this can blow up, I think.
1. The file itself could be encoded wrong.
2. The regex could be having trouble with utf-8, so the term could be unencoded
3. the browser.text could be unencoded.

My first reaction was to add # encoding: utf-8
to the top of each of my files in the project. I should have thought of it earlier — I’ve been having to do this on a lot of projects with Norwegian output lately. Unfortunately that didn’t change the output for the failing scenario.

Next I dug into at the documentation for the ruby encoding class and tried setting default internal and external encoding in the env.rb file:

Encoding.default_internal = Encoding::UTF_8
Encoding.default_external = Encoding::UTF_8

No dice. The output remained the same.

Next, I tried forcing the regex to deal with utf-8 properly (/matcher/u)

Then /^I should see "([^"]*)"$/u do |term|

The output was unchanged.

Reluctantly, I tried what feels like a fairly dirty hack:

browser.text.force_encoding('utf-8').should include(term)

This worked.

So it seems like FireWatir’s text comes back unencoded.

Repeat after me: hypothesis

2010-12-13T16:32:06Z

I have a theory that a butterfly is actually a pegasus that flew too high and died of asphyxiation and then was reborn as a caterpillar.

In short, if it is “just a theory”, then it’s not a theory at all. It might be a hypothesis, or a hunch, or a guess, or sometimes even a conjecture. But certainly not a theory.

And, for the record, evolution is /not/ “just a theory”. It’s well-substantiated, well-supported, and well-documented, explaining scientific observations. I.e. a theory.

Just sayin’.

Order of Callbacks/Validations on Active Record

2010-12-08T16:23:04Z

Every once in a while I find myself wondering about this. So here it is, for the record:

The order of callbacks and validations on a rails active record model is

1. before_validation
2. before_validation, :on => :create
3. after_validation
4. after_validation, :on => :create
5. before_save
6. before_create
7. after_create
8. after_save

ActiveRecord wraps this whole thing in a transaction, so if anything anywhere in this chain fails, the whole thing will be rolled back.

Good to know!

Survival of the fittest

2010-12-07T20:00:00Z

It seems like every time I hear the phrase survival of the fittest it’s used to mean something like the one with the most muscle will win when you duke it out.

Quick reminder: In biology, being fit isn’t about being physically powerful. It’s about having offspring that survive long enough to have offspring.

The phrase “survival of the fittest” is almost redundant, in that it basically states that the genes that survive are the ones that get passed on.

Traditionally this is all about luck and randomness. Humans have recently (in evolutionary terms) added inventiveness into the mix.