Make testing possible with Rails and PostGIS

Inspired by this SO question:

http://stackoverflow.com/questions/8718112/rails-not-fully-compatible-with-postgresqlpostgis-geometry-data-type

I’ve had a lot of trouble recently with Rails and PostGIS. One of the main problem arised with testing. When you run Rails tests (or RSpec for that matter) the test database is always dropped and a new is created. The problem with this is, that the PostGIS extension is not created in this new database. Therefore your schema-load will fail, if you have any postgis-specific fields/indices.

The solution to this is to use a PostgreSQL database template – and of course the activerecord-postgis-adapter found here: https://github.com/dazuma/activerecord-postgis-adapter

First the template:

You need a PostgreSQL template with PostGIS functions support.

Create a template database:

$ psql -U postgres
> CREATE DATABASE template_postgis WITH TEMPLATE=template1 ENCODING='UTF8';
> \c template_postgis;
> CREATE LANGUAGE plpgsql;

Load necessary PostGIS functions into template (your SQL files may be located in a different path, but they come with the PostGIS installation):

$ psql -U postgres -f  /usr/share/postgresql/9.1/contrib/postgis-2.1/postgis.sql template_postgis
$ psql -U postgres -f /usr/share/postgresql/9.1/contrib/postgis-2.1/spatial_ref_sys.sql template_postgis
$ psql -U postgres -f  /usr/share/postgresql/9.1/contrib/postgis-2.1/topology.sql template_postgis

Set database as template and grant permissions:

$ psql -U postgres template_postgis
> UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template_postgis';
> GRANT ALL ON geometry_columns TO PUBLIC;
> GRANT ALL ON spatial_ref_sys TO PUBLIC;
> GRANT ALL ON topology TO PUBLIC;

Now the adapter:

Add this gem to your Gemfile and run bundle:

gem 'activerecord-postgis-adapter'

After that add the adapter and the template to your config/database.yml like this:

development:
  adapter: postgis
  template: template_postgis
  database: mydb_development

test:
  adapter: postgis
  template: template_postgis
  database: mydb_test

And that should do the trick! Test it by running rake db:schema:dump and rake db:test:prepare.

Let me know, if you get any additional problems regarding this.

Make Rails routing case insensitive with route_downcaser gem

A long time ago I solved a nagging issue concerning Rails’ handling of url casing (or rather: non-handling). I posted the resulting code here.

I’ve realised that a lot of people are actually using this solution, so I finally took the time to make it into a fully-tested gem.

The gem is released on RubyGems here.

All documentation (such as is needed) can be found on the gem’s github page.

All you actually need to do is adding the gem to your gemfile, run bundle, and the gem takes care of the rest. No config or initialization code needed, it’s all baked into the gem.

Enjoy! :-)

My favorite Apache setup for a Rails application

You think 1 site equals 1 Apache Virtual Host? Think again, and read on to figure out why not.

For every Rails application that I deploy into production mode, I usually creates 3 Virtual Host entries in Apache. To illustrate, here is the setup for our dog-site www.gipote.dk:

Note: This setup is using Passenger (aka. mod_rails). If you are not already using it for your Rails production environment, you really should do so. It’s the best thing happened since Rails itself.


    ServerName www.gipote.dk

    DocumentRoot /var/www/rails/gipote_production/public

    RailsEnv production



    ServerName  asset0.gipote.dk
    ServerAlias asset1.gipote.dk
    ServerAlias asset2.gipote.dk
    ServerAlias asset3.gipote.dk

    DocumentRoot /var/www/rails/gipote_production/public

    PassengerEnabled off



    ServerName gi-pote.dk
    ServerAlias *.gi-pote.dk
    ServerAlias *.gipote.dk

    RewriteEngine on
    RewriteRule ^(.*) http://www.gipote.dk$1 [R=301]

    PassengerEnabled off

The *=”” inside my virtual host declations is the WordPress SyntaxHighlighter playing tricks on me. It should just be an *

Some explanation is required…

First of all, I want one and only one entry into our site. This originally came from making it easier to setup Google Analytics (you need to tweak the tracking code to support multiple domains). But it seems that it also improves our SEO rankings – so more power to us.

So the first virtual host only accepts www.gipote.dk. The last virtual host accepts all other variations on gipote.dk (except a few – I’ll come to that) as well as an entirely different domain gi-pote.dk. The last virtual host redirects all these requests to www.gipote.dk.

The last virtual host relly must be placed at the bottom, since it contains wildcards. These wildcards – *.gipote.dk – will catch all hostnames not specified in any virtual host above it. Position is vital.

Now to the second virtual host. It requires a bit more explaining…

Internet Explorer and Firefox has a limit when fetching a webpage concerning concurrent connections to the same host. They will only have 2 open connections to a single host at a time. So if you have a page with many external files – or assets (css, js, images), you may find, that it is slow to load, even if you have a powerful server. The bottleneck is at the client this time.

Rails have a fix for this: Every asset specified in the ERB by the asset helpers image_tag(), stylesheet_link_tag(), etc. Can automatically prepend the URL with an asset host. You need to define the asset host in your config/environments/production.rb file:

config.action_controller.asset_host = "http://asset%d.gipote.dk"

The %d is important. This will make Rails’ asset helpers prepend either of


http://asset0.gipote.dk


http://asset1.gipote.dk


http://asset2.gipote.dk


http://asset3.gipote.dk

to any asset url. It will use exactly these 4 variants, and they will be distributed (almost) evenly out on your assets.

So image_tag(‘logo.png’) will become http://asset2.gipote.dk/images/logo.png (or 0, 1, 3).

The cool thing about this is, that the aforementioned browsers will open 2 connections for each host. So instead of fetching 2 assets concurrently, your browser will now fetch 8 at the same time.

So we need our Apache to deal with this. We cannot rely on the last virtual host to do this, because it will simply redirect to www.gipote.dk. And we don’t want to add these 4 hostnames as alias’ to the top virtual host, because we don’t want our users to be able to access our Rails application by any other hostname than www.gipote.dk.

So I create the second virtual host. Put in all 4 hostnames, point it to my Rails public folder. Oh – and for good performance measure, I disable Passenger (PassengerEnabled off). Otherwise the users would still be able to use the hostnames to access our application.

How to make Rails routing case-insensitive

Update 2013-01-06: I have now converted this into a gem: Read more

Update 2011-03-10: Brian Marquis pointed out, that I’ve missed showing a good way to actually load the .rb file containing the middleware. This has now been added to the article.

Update 2010-11-06: Rails 3 has changed its routing mechanism. One of the changes involves which environment variable is used as the source for routing. In Rails 2.3.x it was REQUEST_URI, in Rails 3 it is PATH_INFO. I have changed the middleware code to take care of both versions.

At our dog-site www.gipote.dk we have a shop selling dog-tags. The URL for this shop is www.gipote.dk/hundetegn (hundetegn = dog-tag in danish). However one of our marketing-guys have a tendency to capitalize the word “hundetegn”, so that URL reads www.gipote.dk/Hundetegn (with a capital H).

Rails will yield a 404 NOT FOUND on this, simply because it is case sensitive. The Rails core team doesn’t seem to want this changed, but I found a neat way of doing it myself: Using the new Rack Middleware framework.

First the solution:

Create a file called downcase_route_middleware.rb and put it in RAILS_ROOT/lib or wherever you think middleware files ought to go. Fill it with this piece of code:

class DowncaseRouteMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    # Rails 2.3.x routing use REQUEST_URI
    uri_items = env['REQUEST_URI'].split('?')
    uri_items[0].downcase!
    env['REQUEST_URI'] = uri_items.join('?')

    # Rails 3.x routing use PATH_INFO
    env['PATH_INFO'] = env['PATH_INFO'].downcase

    @app.call(env)
  end
end

Now to telling Rails to use this new middleware class. This differs a bit in Rails 2 and 3.

Adding the middleware in Rails 3.x:

Open config/application.rb and find the lines

module YourAppName
  class Application < Rails::Application
    ..bunch of code..
  end
end

Add the middleware code like this:

module YourAppName
  class Application < Rails::Application
    config.autoload_paths << "#{config.root}/app/middleware"
    config.middleware.use "DowncaseRouteMiddleware"

    ..bunch of code..
  end
end

Adding the middleware in Rails 2.3.x:

Open up config/environment.rb, and find the line

Rails::Initializer.run do |config|

  ...a lot of config code...
end

Add the middleware code, so it looks like this afterwards

Rails::Initializer.run do |config|
  config.autoload_paths << "#{config.root}/app/middleware"
  config.middleware.use "DowncaseRouteMiddleware"

  ...a lot of config code...
end

Update: The line:
config.autoload_paths << "#{config.root}/app/middleware"
is my general way of using middleware. I always places all my middleware classes in a folder called app/middleware. The line above enables Rails to automatically search the autoload-path for a file that matches the middleware class used (i.e. "downcase_route_middleware.rb").

If this is the only middleware class that you use, you can simply do a
require "downcase_route_middleware.rb""
somewhere in your config initializers. I just find the above solution more elegant.

Restart your rails application, and it will now accept all kinds of casing on the routing part.

Note: This code only downcases the part of the URI containing namespace, controller, action, and id. It does not touch the querystring parameters, and for a good reason too: The parameter values could contain some context in their casing.

Now for the explanation:

Rails 2.3 introduced Rack middleware. A lot of people have already explained this concept, so instead of going into details, I suggest that you read Pratik's explanation on Rails Guides and Ryan's Railscast on the subject.

Basically, my middleware class gets access to the environment hash and passes it on to the next middleware class on the stack. In between I get to modify the nessecary data. In this case it's the REQUEST_URI (In Rails 3 it's PATH_INFO), since ActionController::Dispatcher uses the URI from here to determine the correct route. By converting the URI to lowercase, ActionController::Dispatcher gets the correct path no matter the case entered by the user.

Simple Amazon S3 file synchronize for Rails application (without the need of Capistrano)

So you have a nice Rails-app running and you want all your static assets to reside on Amazon S3?

There are other good solutions on the web, but most of what I’ve seen requires that you also use Capistrano for deploying your app. If you have not yet mastered that beast (just like me), you will like this little rake task, that I’ve made.

Just copy/paste this code into your favorite editor, change the 4 parameters in the top, and save it in lib/tasks/aws.rake.

Then you can do this:

# rake aws:sync

This will upload all folders and files from your public folder into your Amazon S3 bucket.

The script creates a file called .aws_cache in the root of your application folder. This is used so that additional calls to rake aws:sync only uploads new and modified files.

If you have already uploaded your files before applying this script, you can start by calling

# rake aws:build_cache

This will build the local cachefiles without uploading anything to Amazon S3.

Here’s the code:

require 'find'
require 'digest/md5'
require 'yaml'

#########################################################
# Configuration - these lines are all you need to edit
ACCESS_KEY_ID     = "your-own-aws-access-key"
SECRET_ACCESS_KEY = "and-the-secret-key"
BUCKET             = "name-of-your-bucket"

# If you have any subfolders inside "public", that you do not want to place on AWS, list them here
IGNORE_FOLDERS    = %w(upload UserFiles videos)
#########################################################

class AwsCache
  def initialize
    @filename = ".aws_cache"
    load
  end

  def clear
    @cache = {}
  end

  def load
    clear
    if File.exists?(@filename)
      @cache = YAML::load_file(@filename)
    end
  end

  def save
    File.open(@filename, "w") do |f|
      f.puts @cache.to_yaml
    end
  end

  def add(path)
    puts "add to cache: #{path}"
    @cache[path] = checksum(path)
  end

  def equal?(path)
    c = checksum(path)
    @cache[path] == c
  end

  protected
  def checksum(path)
    Digest::MD5.file(path).hexdigest
  end
end

namespace :aws do
  desc "Synchronize public folder"
  task :sync => :environment do
    init

    AWS::S3::Base.establish_connection!(
      :access_key_id     => ACCESS_KEY_ID,
      :secret_access_key => SECRET_ACCESS_KEY
    )

    bucket = AWS::S3::Bucket.find(@bucket)

    loop_folder()

    @aws_cache.save
  end

  desc "Rebuild local cache file"
  task :build_cache => :environment do
    init

    absolute_folder = File.join(@base_folder, "")

    Find.find(absolute_folder) do |path|
      if FileTest.directory?(path)
        if File.basename(path)[0] == ?.
          Find.prune
        elsif IGNORE_FOLDERS.include?(File.basename(path))
          Find.prune
        end
      else
        @aws_cache.add(path)
      end
    end

    @aws_cache.save
  end
end

def init
  @base_folder = File.join(RAILS_ROOT, "public")
  @bucket      = BUCKET
  @aws_cache   = AwsCache.new
end

def loop_folder(folder = "")
  absolute_folder = File.join(@base_folder, folder)

  total_size = 0

  Find.find(absolute_folder) do |path|
    if FileTest.directory?(path)
      if File.basename(path)[0] == ?.
        Find.prune
      elsif IGNORE_FOLDERS.include?(File.basename(path))
        Find.prune
      else
        next
      end
    else
      s3_store(path)
    end
  end
end

def s3_store(path)
  s3_path = path.gsub("#{@base_folder}/", '')

  if transfer?(path)
    AWS::S3::S3Object.store(
      s3_path,
      open(path),
      @bucket,
      :access => :public_read)

    puts "Stored into AWS: #{AWS::S3::S3Object.url_for(s3_path, @bucket)[/[^?]+/]}"
  end
end

def transfer?(path)
  return false if @aws_cache.equal?(path)

  @aws_cache.add(path)
  return true
end

Feel free to contact me, if you have any questions, problems, suggestions for improvement.

Test test og atter test

Jeg er nu endelig kommet på den rette vej med TDD. Jeg må jo indrømme, at TDD for mig indtil nu har været “kode først, tests bagefter”, men jeg så forleden lige pludselig “lyset”. Det har også kun taget 2¾ år med Rails at nå dertil… ;-)

Hvad gjorde så udslaget for mig? Jo det var faktisk, da jeg læste denne blog http://code.isdangero.us/posts/My-test-driven-Ruby-setup

Jeg kunne godt lide syntaksen i Shoulda. Men det var især Machinist og den lette måde at lave test-data på, som gav mig en “aha-oplevelse”. Fixtures er besværlige at lave, og det er nemt at knække eksisterende tests, når man tilføjer flere tests. Machinist ændrer på alt det, og det gik pludselig op for mig, at det netop var dét, der var bremsen for min omvenden.

Men for pokker hvor er der pludselig mange forskellige frameworks/ libraries til TDD/BDD. Det er jo nærmest en jungle. Det er sjovt, at hvor de fleste bare accepterer ActiveRecord og de øvrige konventioner i Rails, så bruger rigtig mange tilsyneladende alt muligt andet end Rails’ egent test-suite.

Hvilken kombi bruger i og hvorfor? Hvor nemt har i ved at teste først og kode bagefter? Jeg er gammel i gårde, og jeg må indrømme, at det er en af de steder, hvor jeg har svært ved at ændre gamle vaner.

- Carsten

Thinking about searching in your app?

I’ll cut to the chase. There is really only one good search engine for Rails applications at the moment of this writing.

It’s called ThinkingSphinx, it’s fast, it’s reliable, and it has the easiest Rails implementation, that I have seen so far.

There is really no need for me to do a tutorial here, since Pat Allan has already done a great one himself. Read it here.

I use ThinkingSphinx in production on my Gipote site. Before that I’ve been with Ferret (not good in a multi-mongrel production site), and UltraSphinx (gave me great problems when indexing multiple models). After using ThinkingSphinx as my search engine, all pain has gone.

Hooray – now full-time on Rails

I have decided to sell my company and get a job at a company developing solely in Ruby on Rails.

This is really the best decision, that I’ve done for myself AND my family for a very long time. The career as self-employed was really starting to tear me down.

So now – almost 2 years since I complained about the desire to work only with Rails but having the economical need to do PHP – I have finally shedded the PHP load and am now a full-time Rails developer.

Dear PHP: Thank you for 9 great years. It has been fun, but now I really must move on.

At the same time, I decided to change the name of my blog. I will be having a lot of travelling by train each day, and although I will do a lot of coding on these trips, it is also a perfect place, to write down the thoughts and ideas coming up during the day.

Tabs be damned

- at least that seems to be a clear oppinion of the Rails community.

Okay so I lied to you in the last post. I’ve been working on my first real Rails project for quite some time now, so instead of a realtime diary, I will be reflecting upon the major issues, that I went through (eg. “how to make a multiple-page wizard-like create controller”). The reason being, that I haven’t been allowed to speak in public about the project until now – we will release it in the coming week.

But I will break the cronology by starting with one of my latest issues:

Ruby on Rails does not allow tab characters for indentation in RHTML templates

Have you ever tried to get a wierd “SyntaxError Exception” when calling a page in your Rails application? There is nothing wrong with the inlined Ruby code.

It turns out, that if you indent your RHTML code with tab characters instead of spaces, the ActionViewer will report a compile error when evaluating your template.

I know, accept, and also appreciate, that Rails is “oppinionated” software. But this is really ridiculous. IMHO tabs are the only way to go – combined of course with spaces for alignment.

This has been an issue at least since Rails 1.1 (I haven’t been on Rails before that), and I found somebody mentioning a workaround then.

So I patched Rails and worked happily ever after – that is until yesterday, when I decided I wanted to release the app with Rails 1.2 instead of 1.1 (better to work out the knots before going live).

I was slammed with RHTML Syntax Errors again. And I had completely forgotten about this little patch, which was, of course, now overruled by the new version of Rails.

It turned out (after much debugging), that the ActionView has been recoded quite a lot, so the previous patch could not be applied in the same place.

But here is what you should do:

1) In Ruby’s lib folder, find the file:
rubygems1.8gemsactionpack-1.13.1libaction_viewbase.rb

2) From line 504:

def compile_template(extension, template, file_name, local_assigns)
render_symbol = assign_method_name(extension, template, file_name)
render_source = create_template_source(extension, template, render_symbol, local_assigns.keys)

add these two lines just in the top of the method:

def compile_template(extension, template, file_name, local_assigns)
# convert all tab-characters to space-indentation (4 spaces)
template = template.gsub(/t/, ” “) unless template.nil?

render_symbol = assign_method_name(extension, template, file_name)
render_source = create_template_source(extension, template, render_symbol, local_assigns.keys)

Now I only need to find out, how I can format my code in Blogger… ;-)