Make testing possible with Rails and PostGIS

Inspired by this SO question:

http://stackoverflow.com/questions/8718112/rails-not-fully-compatible-with-postgresqlpostgis-geometry-data-type

I’ve had a lot of trouble recently with Rails and PostGIS. One of the main problem arised with testing. When you run Rails tests (or RSpec for that matter) the test database is always dropped and a new is created. The problem with this is, that the PostGIS extension is not created in this new database. Therefore your schema-load will fail, if you have any postgis-specific fields/indices.

The solution to this is to use a PostgreSQL database template – and of course the activerecord-postgis-adapter found here: https://github.com/dazuma/activerecord-postgis-adapter

First the template:

You need a PostgreSQL template with PostGIS functions support.

Create a template database:

$ psql -U postgres
> CREATE DATABASE template_postgis WITH TEMPLATE=template1 ENCODING='UTF8';
> \c template_postgis;
> CREATE LANGUAGE plpgsql;

Load necessary PostGIS functions into template (your SQL files may be located in a different path, but they come with the PostGIS installation):

$ psql -U postgres -f  /usr/share/postgresql/9.1/contrib/postgis-2.1/postgis.sql template_postgis
$ psql -U postgres -f /usr/share/postgresql/9.1/contrib/postgis-2.1/spatial_ref_sys.sql template_postgis
$ psql -U postgres -f  /usr/share/postgresql/9.1/contrib/postgis-2.1/topology.sql template_postgis

Set database as template and grant permissions:

$ psql -U postgres template_postgis
> UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template_postgis';
> GRANT ALL ON geometry_columns TO PUBLIC;
> GRANT ALL ON spatial_ref_sys TO PUBLIC;
> GRANT ALL ON topology TO PUBLIC;

Now the adapter:

Add this gem to your Gemfile and run bundle:

gem 'activerecord-postgis-adapter'

After that add the adapter and the template to your config/database.yml like this:

development:
  adapter: postgis
  template: template_postgis
  database: mydb_development

test:
  adapter: postgis
  template: template_postgis
  database: mydb_test

And that should do the trick! Test it by running rake db:schema:dump and rake db:test:prepare.

Let me know, if you get any additional problems regarding this.

Make Rails routing case insensitive with route_downcaser gem

A long time ago I solved a nagging issue concerning Rails’ handling of url casing (or rather: non-handling). I posted the resulting code here.

I’ve realised that a lot of people are actually using this solution, so I finally took the time to make it into a fully-tested gem.

The gem is released on RubyGems here.

All documentation (such as is needed) can be found on the gem’s github page.

All you actually need to do is adding the gem to your gemfile, run bundle, and the gem takes care of the rest. No config or initialization code needed, it’s all baked into the gem.

Enjoy! :-)

Designing with Compass – my new best friend with mobile applications

Ruby on Rails has always been marketed as “Webdevelopment that doesn’t hurt”, and a lot of this comes from standing on the shoulders of giants.

One of the more recent pair of shoulders to pop up is Compass. I would say that Compass is a natural development, stemming from Haml and Sass. It has made stylesheet a more natural part of Rails application development.

While a lot of tools exists for creating “html-generated applications”, stylesheets have always been left a bit in the cold. Sass helped a bit by adding variables and partials, but it still didn’t feel like programming.

Compass adds two important things:

  • mixins which enables you to create behaviors that you apply to several css classes.
  • functions which can be used to generate css values based on parameters

To show why this is great, here’s an example of doing grid layout the Right Way(tm) – and The-Way-You-Used-To-Do-It…

Grid layouts is usually coupled with frameworks like Blueprint, 960 and others. One of the problems with this is, that you do not get semantic markup in your HTML. Instead you get all kinds of layout classes. Here’s an simple example of a typical Blueprint HTML layout:

!!!
%head
%body
  .container
    .main.span-18
      ...a lot of content...
    .sideinfo.span-6.last
      ... links etc. in the sidebar...

and in HTML for those of you who have not tried Haml yet (you really should):

<html>
<head></head>
<body>
  
...a lot of content...
... links etc. in the sidebar...
</body> </html>

Why is this bad? Well if you are only developing for traditional computers, you’re fine. But what if you want to use the same webapplication for Android or iOS devices? HTML5 has a nifty feature for stylesheet selection based on the clients screen size:



etc...

This is pretty cool – but then all the span-classes from our Blueprint example suddenly gets out of context.

Now back to why Compass is cool

Compass makes grid-design easy without using anything but your semantic markup. Here’s the same layout done with compass (again in Haml and HTML) – explanations can be found on Compass’ homepage.

!!!
%head
  ...stylesheet inclusion...
%body
  .container
    .main
      ...a lot of content...
    .sideinfo
      ... links etc. in the sidebar...
<html>
<head>  ...stylesheet inclusion...</head>
<body>
  
...a lot of content...
... links etc. in the sidebar...
</body> </html>

IMHO much prettier – and much easier to mould with different stylesheets. Here’s what it takes in Compass using mixins and functions:

@import "compass"

$blueprint_grid_columns: 24;
$blueprint_grid_width: 40px;
@import "blueprint"

.main
  +column(18)

.sideinfo
  +column(6, true)

Now if you have a 320px device (phone) you could switch to a much simpler stylesheet – without worrying about any layout-specific HTML markup.

Avoid html escaping I18n strings in Rails 3

Update: Be careful with this method, if you have string-interpolation with user input. It might be the cause of XSS. I’ll cough up a better method, that can handle cases with and without interpolation

With Rails 3 comes a shift in who is responsible for escaping output. You no longer need to write <%= h(@some_value)%>, Rails will automatically escape every string unless you mark it html-safe.

Unfortunately this also goes for I18n strings. In my case I have a lot of strings containing HTML entities like &oslash; which is now escaped by Rails when I write <%=t(:some_key)%> and therefore “Søg” is now displayed as “S&oslash;g”. Not good.

A tiny fix for this:

module I18n
  class << self
    alias :prev_translate :translate
    def translate(*args)
      prev_translate(*args).html_safe
    end
  end
end

This ensures that I18n output is marked as html-safe. Just place it as a initializer and your translations will work as before.

Trouble with Net::HTTP in Ruby 1.8.7 (released with Ubuntu 10.04)

I stumbled upon an error trying to work with acts_as_solr just after upgrading to Ubuntu 10.04. When trying to start the solr-daemon with

rake solr:start

i suddenly got an error not seen before:

undefined method `closed?' for nil:NilClass
/usr/lib/ruby/1.8/net/http.rb:1060:in `request'
/usr/lib/ruby/1.8/net/http.rb:962:in `request_head'

As the error hints, it’s something to do with the net/http library. After some googling I found out, that it is because Ubuntu 10.04 ships with Ruby 1.8.7, and that this particular version has a bug in the net/http library. Here’s the bug-report and the patch that will help:

http://redmine.ruby-lang.org/issues/show/2708

Hopefully this patch will soon find its way to Canonical’s updates.

My favorite Apache setup for a Rails application

You think 1 site equals 1 Apache Virtual Host? Think again, and read on to figure out why not.

For every Rails application that I deploy into production mode, I usually creates 3 Virtual Host entries in Apache. To illustrate, here is the setup for our dog-site www.gipote.dk:

Note: This setup is using Passenger (aka. mod_rails). If you are not already using it for your Rails production environment, you really should do so. It’s the best thing happened since Rails itself.


    ServerName www.gipote.dk

    DocumentRoot /var/www/rails/gipote_production/public

    RailsEnv production



    ServerName  asset0.gipote.dk
    ServerAlias asset1.gipote.dk
    ServerAlias asset2.gipote.dk
    ServerAlias asset3.gipote.dk

    DocumentRoot /var/www/rails/gipote_production/public

    PassengerEnabled off



    ServerName gi-pote.dk
    ServerAlias *.gi-pote.dk
    ServerAlias *.gipote.dk

    RewriteEngine on
    RewriteRule ^(.*) http://www.gipote.dk$1 [R=301]

    PassengerEnabled off

The *=”” inside my virtual host declations is the WordPress SyntaxHighlighter playing tricks on me. It should just be an *

Some explanation is required…

First of all, I want one and only one entry into our site. This originally came from making it easier to setup Google Analytics (you need to tweak the tracking code to support multiple domains). But it seems that it also improves our SEO rankings – so more power to us.

So the first virtual host only accepts www.gipote.dk. The last virtual host accepts all other variations on gipote.dk (except a few – I’ll come to that) as well as an entirely different domain gi-pote.dk. The last virtual host redirects all these requests to www.gipote.dk.

The last virtual host relly must be placed at the bottom, since it contains wildcards. These wildcards – *.gipote.dk – will catch all hostnames not specified in any virtual host above it. Position is vital.

Now to the second virtual host. It requires a bit more explaining…

Internet Explorer and Firefox has a limit when fetching a webpage concerning concurrent connections to the same host. They will only have 2 open connections to a single host at a time. So if you have a page with many external files – or assets (css, js, images), you may find, that it is slow to load, even if you have a powerful server. The bottleneck is at the client this time.

Rails have a fix for this: Every asset specified in the ERB by the asset helpers image_tag(), stylesheet_link_tag(), etc. Can automatically prepend the URL with an asset host. You need to define the asset host in your config/environments/production.rb file:

config.action_controller.asset_host = "http://asset%d.gipote.dk"

The %d is important. This will make Rails’ asset helpers prepend either of


http://asset0.gipote.dk


http://asset1.gipote.dk


http://asset2.gipote.dk


http://asset3.gipote.dk

to any asset url. It will use exactly these 4 variants, and they will be distributed (almost) evenly out on your assets.

So image_tag(‘logo.png’) will become http://asset2.gipote.dk/images/logo.png (or 0, 1, 3).

The cool thing about this is, that the aforementioned browsers will open 2 connections for each host. So instead of fetching 2 assets concurrently, your browser will now fetch 8 at the same time.

So we need our Apache to deal with this. We cannot rely on the last virtual host to do this, because it will simply redirect to www.gipote.dk. And we don’t want to add these 4 hostnames as alias’ to the top virtual host, because we don’t want our users to be able to access our Rails application by any other hostname than www.gipote.dk.

So I create the second virtual host. Put in all 4 hostnames, point it to my Rails public folder. Oh – and for good performance measure, I disable Passenger (PassengerEnabled off). Otherwise the users would still be able to use the hostnames to access our application.

How to make Rails routing case-insensitive

Update 2013-01-06: I have now converted this into a gem: Read more

Update 2011-03-10: Brian Marquis pointed out, that I’ve missed showing a good way to actually load the .rb file containing the middleware. This has now been added to the article.

Update 2010-11-06: Rails 3 has changed its routing mechanism. One of the changes involves which environment variable is used as the source for routing. In Rails 2.3.x it was REQUEST_URI, in Rails 3 it is PATH_INFO. I have changed the middleware code to take care of both versions.

At our dog-site www.gipote.dk we have a shop selling dog-tags. The URL for this shop is www.gipote.dk/hundetegn (hundetegn = dog-tag in danish). However one of our marketing-guys have a tendency to capitalize the word “hundetegn”, so that URL reads www.gipote.dk/Hundetegn (with a capital H).

Rails will yield a 404 NOT FOUND on this, simply because it is case sensitive. The Rails core team doesn’t seem to want this changed, but I found a neat way of doing it myself: Using the new Rack Middleware framework.

First the solution:

Create a file called downcase_route_middleware.rb and put it in RAILS_ROOT/lib or wherever you think middleware files ought to go. Fill it with this piece of code:

class DowncaseRouteMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    # Rails 2.3.x routing use REQUEST_URI
    uri_items = env['REQUEST_URI'].split('?')
    uri_items[0].downcase!
    env['REQUEST_URI'] = uri_items.join('?')

    # Rails 3.x routing use PATH_INFO
    env['PATH_INFO'] = env['PATH_INFO'].downcase

    @app.call(env)
  end
end

Now to telling Rails to use this new middleware class. This differs a bit in Rails 2 and 3.

Adding the middleware in Rails 3.x:

Open config/application.rb and find the lines

module YourAppName
  class Application < Rails::Application
    ..bunch of code..
  end
end

Add the middleware code like this:

module YourAppName
  class Application < Rails::Application
    config.autoload_paths << "#{config.root}/app/middleware"
    config.middleware.use "DowncaseRouteMiddleware"

    ..bunch of code..
  end
end

Adding the middleware in Rails 2.3.x:

Open up config/environment.rb, and find the line

Rails::Initializer.run do |config|

  ...a lot of config code...
end

Add the middleware code, so it looks like this afterwards

Rails::Initializer.run do |config|
  config.autoload_paths << "#{config.root}/app/middleware"
  config.middleware.use "DowncaseRouteMiddleware"

  ...a lot of config code...
end

Update: The line:
config.autoload_paths << "#{config.root}/app/middleware"
is my general way of using middleware. I always places all my middleware classes in a folder called app/middleware. The line above enables Rails to automatically search the autoload-path for a file that matches the middleware class used (i.e. "downcase_route_middleware.rb").

If this is the only middleware class that you use, you can simply do a
require "downcase_route_middleware.rb""
somewhere in your config initializers. I just find the above solution more elegant.

Restart your rails application, and it will now accept all kinds of casing on the routing part.

Note: This code only downcases the part of the URI containing namespace, controller, action, and id. It does not touch the querystring parameters, and for a good reason too: The parameter values could contain some context in their casing.

Now for the explanation:

Rails 2.3 introduced Rack middleware. A lot of people have already explained this concept, so instead of going into details, I suggest that you read Pratik's explanation on Rails Guides and Ryan's Railscast on the subject.

Basically, my middleware class gets access to the environment hash and passes it on to the next middleware class on the stack. In between I get to modify the nessecary data. In this case it's the REQUEST_URI (In Rails 3 it's PATH_INFO), since ActionController::Dispatcher uses the URI from here to determine the correct route. By converting the URI to lowercase, ActionController::Dispatcher gets the correct path no matter the case entered by the user.

The woes of libodbc-ruby1.8 and Debian + Ubuntu

Update 05-05-2010: This issue still exists on the new Ubuntu 10.04 LTS (Lucid Lynx). The dist-upgrade will override any version locks that you have made on libodbc-ruby1.8 and install version 0.9997-2 (and lock it). So after a dist-upgrade you will have to remove libodbc-ruby1.8 and install the older version again.

I have the dubious honor of developing on a Rails application, that runs on top of Microsoft SQL Server. This has given some headaches at our most recent system upgrades.

My development machine runs Ubuntu 9.10 (Karmic Koala). Our servers run Debian 5.0 (Lenny).

If you run any of these OS versions and encounter an error like the following:

dbd-odbc-0.2.4/lib/dbd/odbc/driver.rb:36:in `connect':DBI::DatabaseError: INTERN (0) [RubyODBC]Cannot allocate SQLHENV

then help can be found here.

The problem on both OS’es lies in the package libodbc-ruby1.8. But it is actually stranger than you might expect, which I will explain below.

Ubuntu first

We’ll start off with Ubuntu. Fire up Synaptic Package Manger and search for libodbc-ruby1.8. You will see, that the distribution package is versioned 0.9997-2. You need to uninstall this and install version 0.9995-1 instead. You can download version 0.9995-1 here: http://packages.ubuntu.com/hu/hardy/i386/libodbc-ruby1.8/download

Now install it with apt-get and… problem solved. Please note however that your automatic package update will revert to version 0.9997-2 unless you uncheck it before running the update.

Update 19-02-2010: This is also an issue on the 64-bit version of Ubuntu 9.10. Here’s a link where you can download the 0.9995-1 .deb package for both 32-bit AND 64-bit. Thanks Goran.
http://mirrors.kernel.org/ubuntu/pool/universe/libo/libodbc-ruby

Debian next

On Debian it’s the other way around. Debian comes installed with libodbc-ruby1.8 version 0.9995-1 and you need to upgrade this to 0.9997-2. This package is currently only found in unstable, but you can download it here: http://packages.debian.org/sid/i386/libodbc-ruby1.8/download

Install it with apt-get, overriding the previous package.

Hope this saves you from the days of work that I invested in this particular oddity. :-)

Reclaim control of your application with Cucumber

As I’ve written earlier(danish), I’ve arrived rather late to the test-driven development-train. As a result I have a large application, developed in Rails, but without utilizing the test framework. Granted I have lately added some unit-tests, but in the whole, I would deem the application uncovered by tests.

Such a beast is difficult to handle. To upgrade it to support the latest Rails version (2.3.5 in this time of writing) is like jumping of a plane and hope to catch a floating parachute on your way down. On the other hand, writing unit-tests and functional tests to cover all current functionality is a huge task, that is discouraging simply by its presence. To put it short: The application has grown beyond my control.

So how to tackle this situation? I want the entire application (or most of at least) covered by test, but I will not be able to spare the next 3 months doing nothing but writing unit and functional tests.

Enter Cucumber.

With Cucumber I am able to create happy paths for all the use cases in my application. I know how each area of the application is supposed to be used, and by outlining these uses in Cucumber features and scenarios, I get a basic coverage of all functionality.

This is a very top-down approach. I don’t get to test each small functionality separately. And even the Cucumber guys encourage coders to write more than just happy paths (adding also scenarios that are supposed to test for error messages). But at least, if something breaks, I will know about it. So when I change something in my application, I run all my cucumber features. If one of the scenarios fail, I can now focus on writing more detailed tests for this particular part of the application.

A big thanks to Aslak Hellesøy and friends for helping me reclaim control of my application!

Simple Amazon S3 file synchronize for Rails application (without the need of Capistrano)

So you have a nice Rails-app running and you want all your static assets to reside on Amazon S3?

There are other good solutions on the web, but most of what I’ve seen requires that you also use Capistrano for deploying your app. If you have not yet mastered that beast (just like me), you will like this little rake task, that I’ve made.

Just copy/paste this code into your favorite editor, change the 4 parameters in the top, and save it in lib/tasks/aws.rake.

Then you can do this:

# rake aws:sync

This will upload all folders and files from your public folder into your Amazon S3 bucket.

The script creates a file called .aws_cache in the root of your application folder. This is used so that additional calls to rake aws:sync only uploads new and modified files.

If you have already uploaded your files before applying this script, you can start by calling

# rake aws:build_cache

This will build the local cachefiles without uploading anything to Amazon S3.

Here’s the code:

require 'find'
require 'digest/md5'
require 'yaml'

#########################################################
# Configuration - these lines are all you need to edit
ACCESS_KEY_ID     = "your-own-aws-access-key"
SECRET_ACCESS_KEY = "and-the-secret-key"
BUCKET             = "name-of-your-bucket"

# If you have any subfolders inside "public", that you do not want to place on AWS, list them here
IGNORE_FOLDERS    = %w(upload UserFiles videos)
#########################################################

class AwsCache
  def initialize
    @filename = ".aws_cache"
    load
  end

  def clear
    @cache = {}
  end

  def load
    clear
    if File.exists?(@filename)
      @cache = YAML::load_file(@filename)
    end
  end

  def save
    File.open(@filename, "w") do |f|
      f.puts @cache.to_yaml
    end
  end

  def add(path)
    puts "add to cache: #{path}"
    @cache[path] = checksum(path)
  end

  def equal?(path)
    c = checksum(path)
    @cache[path] == c
  end

  protected
  def checksum(path)
    Digest::MD5.file(path).hexdigest
  end
end

namespace :aws do
  desc "Synchronize public folder"
  task :sync => :environment do
    init

    AWS::S3::Base.establish_connection!(
      :access_key_id     => ACCESS_KEY_ID,
      :secret_access_key => SECRET_ACCESS_KEY
    )

    bucket = AWS::S3::Bucket.find(@bucket)

    loop_folder()

    @aws_cache.save
  end

  desc "Rebuild local cache file"
  task :build_cache => :environment do
    init

    absolute_folder = File.join(@base_folder, "")

    Find.find(absolute_folder) do |path|
      if FileTest.directory?(path)
        if File.basename(path)[0] == ?.
          Find.prune
        elsif IGNORE_FOLDERS.include?(File.basename(path))
          Find.prune
        end
      else
        @aws_cache.add(path)
      end
    end

    @aws_cache.save
  end
end

def init
  @base_folder = File.join(RAILS_ROOT, "public")
  @bucket      = BUCKET
  @aws_cache   = AwsCache.new
end

def loop_folder(folder = "")
  absolute_folder = File.join(@base_folder, folder)

  total_size = 0

  Find.find(absolute_folder) do |path|
    if FileTest.directory?(path)
      if File.basename(path)[0] == ?.
        Find.prune
      elsif IGNORE_FOLDERS.include?(File.basename(path))
        Find.prune
      else
        next
      end
    else
      s3_store(path)
    end
  end
end

def s3_store(path)
  s3_path = path.gsub("#{@base_folder}/", '')

  if transfer?(path)
    AWS::S3::S3Object.store(
      s3_path,
      open(path),
      @bucket,
      :access => :public_read)

    puts "Stored into AWS: #{AWS::S3::S3Object.url_for(s3_path, @bucket)[/[^?]+/]}"
  end
end

def transfer?(path)
  return false if @aws_cache.equal?(path)

  @aws_cache.add(path)
  return true
end

Feel free to contact me, if you have any questions, problems, suggestions for improvement.