caboose

With the help of Jamie van Dyke at Parfait and Scott Chacon at GitHub, I'm pleased to announce Acts As Git (no, I don't like the name either). It's a simple plugin which stores all changes you make to a text field in a git repository. This is ideal for something like a git-backed wiki.

Look at it here: github or check it out from

git://github.com/courtenay/acts_like_git.git

From the README:

ALG automagically saves the history of a given text or string field. It sits over the top of an ActiveRecord model; after a value is committed to the database, the plugin writes the new value to a text file and commits it to a git repository. This way you get all the advantages of using Git as version-control.

Usage:

class Post < ActiveRecord::Base
  versioning(:title) do |version|
    version.repository = '/home/git/repositories/postal.git'
    version.message = lambda { |post| "Committed by #{post.author.name}" }
  end
end

To view the complete list of changes:

>> @post = Post.find 15
<Post:15>
>> @post.title
=> 'Freddy'
>> @post.history(:title)
=> ['Joe', 'Frank', 'Freddy]
>> @post.log
=> ['bfec2f69e270d2d02de4e8c7a4eb2bd0f132bdbb', '643deb45c12982dde75ba71657792a2dbdda83e6', 
'1ce6c7368219db7698f4acc3417e656510b4138d']
>> @post.revert_to '1ce6c7368219db7698f4acc3417e656510b4138d'
>> @post.title
=> 'Joe'

It uses the excellent Grit library, and doesn't actually have a checked-out repository. The latest version of your data is still stored in the database. You can actually clone this repo and view the changes; pushing back to it won't do anything useful.

I’m putting the final touches on a super-sweet versioning plugin, and I’ve discovered that we’re using several different metaphors for configuring the plugin options. I’d like to get some opinions/feedback on your preferred style.

The DSL

Using a DSL and passing blocks in which get instance evalled. I’m normally very scathing of DSLs; I think that they’re Yet Another Language for people to learn to use – it’s usually your very own write-only syntax – but it’s been super-fun implementing the backend to this.

class Monkey < ActiveRecord::Base
  versioning do
    author do
      name { user.current.name }
      message { "Commited via #{name}" }
    end
    repository "Joe's DataStore" 
  end

Hashes

This seems to be the Rails plugin default:

class Monkey < ActiveRecord::Base
  versioning :author => { :name => lambda{ |u| user.current.name } }, :repository => "Joe's DataStore" 
end

Class vars / methods

Easy to monkeypatch later

class Monkey < ActiveRecord::Base
   will_version
   @@version_repository = "Joe's DataStory" 
   def version_author
     current_name
   end
end

Are there others? Which do you prefer? Currently I’m using all three in this one plugin, and it’s very un-awesome.

I sat down with David Chelimsky at Rubyconf today to talk about rSpec and an interesting topic came up.

In my mind, there are two reasons to use a mock object: first, when you’re developing TDD style, you physically don’t have the objects yet; and second, so that you can tightly focus your unit tests. Maybe, these two different purposes should use a different mechanism.

His question to me then was, “Do you replace your mocks with the real objects after you’ve implemented those objects?”. I guess I hadn’t thought about that before. Do you? If so, how do you handle the extra complexity, maintaining sane associations and valid data?

We’re launching a new service at work in the next week or so that involves me looking through a lot of job applications: resumes and sample code.

I’d like to tell people right now, upfront, if you’re applying for a Ruby or Rails job, for anyone, there are a few ways of ensuring you get called back. They’re probably fairly simple.

Send some sample code, maybe a link to a project on Github, or a snippet of work you’ve done. Make sure you send the tests for the code. Any tests would be good, and you get bonus points for good tests. If you don’t have any tests, write them.

Don’t worry too much about sending some crazy complex code. Maybe some polymorphic associations (models), some ajax (views), a knowledge of the whole stack (simple controllers), some nested resources. Write a simple todo list application.

It’s not just a silly philosophy. Writing tests – hell, submitting tests with your job application’s code – shows that you’ve actually thought about the code, and that it actually works. You’ve permutated and permeated through the logic, actually think about the various ramifications of the design decisions in the code itself.

Just the pure act of sending tests with your sample code will put you above 90% of applicants, I promise.

...for new projects.

We upgraded the gems for one of our client projects, and the auto-loading / config.gems managed to completely break all our other projects, requiring upgrades, which caused weird breakages in weird places in some of the specs.

The app would refuse to deploy (rake tmp:create failed, because lib/tasks/rspec.rake was being loaded, and spec wasn't installed on the server). The annoying thing was that just having whatever.11 installed (I don't know the exact version) broke older apps on whatever.4 or whatever.0.2. .. so those had to be upgraded too. We wasted a day or two (three, maybe four developers) which equates to several thousand dollars in wasteage. It was also really infuriating -- the culmination of a few years of frustration of rSpec's weirdnesses.

After that, I found that some of the specs had never run (who knows why). It stopped reading spec.opts and started doing some weirdness with pending options. Finally, Rick just snapped, threw out rSpec and his Model Stubbing library, and now we're playing with a combination of rr, context, and matchy, trying to get a feel for a decent workflow again. It's sad and maybe a bit exciting to be on the edge.

What are you testing with?

A few years ago I wrote a simple addition to ActiveRecord that does two things: it chops out the eager loading "t1_t2 AS foo", and it shows the number of records returned for every query you run against the database. You can view the file here

Today I was profiling a site and wanted to quickly find the slow database queries, but didn't have access to mysql's config directly, so I patched that file above to record all queries over 500ms and save it to a log file. I'll warn you now, it ain't pretty, but it works pretty well.

Here's how it works: First, throw this in a file in config/initializers. I open up the rails abstract adapter

module ActiveRecord
  module ConnectionAdapters
    class AbstractAdapter

And add in a new logger.

        def slow_query; 0.5; end # number of seconds
        def slow_query_logger
          @slow_query_logger ||= Logger.new("log/slow_queries.log")
        end

Ideally of course this would all be configurable.

Next, I copy the logging code out of the latest ActiveRecord, and patch it to return the number of records. This is a bit of a hack, too, but we can either look at "num_rows" from the resultset or the actual size of an array.

              s = result && (result.respond_to?(:num_rows) ? result.num_rows : \
                 (result.respond_to?(:size) ? result.size : 0)) || 0

Finally, I rewrite the actual log method so that it checks the benchmark against our threshold

        def log_info(sql, name, runtime, result_size = 0)
          if runtime > slow_query && slow_query_logger
            slow_query_logger.debug "Slow query: (#{runtime}) [#{result_size}] #{sql}"
          end

And add the number of results to the regular rails log, while snipping out the annoying eager-loading code.

          if @logger && @logger.debug?
            if name =~ /Load Including Associations$/
              sql = sql.scan(/SELECT /).to_s + ' ...<snip>... ' + sql.scan(/(FROM .*)$/).to_s
            end

            name = "#{name.nil? ? "SQL" : name} (#{sprintf("%f", runtime)}) [#{result_size.to_i}]"
            @logger.debug format_log_entry(name, sql.squeeze(' '))
          end
        end

Here's the full file.

module ActiveRecord
  module ConnectionAdapters # :nodoc:
    class AbstractAdapter
      protected
        # todo: config this
        def slow_query; 0.5; end
        def slow_query_logger
          @slow_query_logger ||= Logger.new("log/slow_queries.log")
        end

        alias_method :old_log, :log

        def log(sql, name, &block)
          if block_given?
            #if @logger and @logger.level <= Logger::INFO
              result = nil
              seconds = Benchmark.realtime { result = yield }
              @runtime += seconds
              s = result && (result.respond_to?(:num_rows) ? result.num_rows : \
                 (result.respond_to?(:size) ? result.size : 0)) || 0 
              log_info(sql, name, seconds, s)
              return result
            #end
          else
            log_info(sql, name, 0, 0)
            nil
          end
          # old_log(sql, name) { yield }
        rescue Exception => e
          @last_verification = 0
          message = "#{e.class.name}: #{e.message}: #{sql}"
          log_info(message, name, 0)
          raise ActiveRecord::StatementInvalid, message
        end

        alias_method :old_log_info, :log_info
        def log_info(sql, name, runtime, result_size = 0)
          if runtime > slow_query && slow_query_logger
            slow_query_logger.debug "Slow query: (#{runtime}) [#{result_size}] #{sql}"
          end
          if @logger && @logger.debug?
            if name =~ /Load Including Associations$/
              sql = sql.scan(/SELECT /).to_s + ' ...<snip>... ' + sql.scan(/(FROM .*)$/).to_s
            end

            name = "#{name.nil? ? "SQL" : name} (#{sprintf("%f", runtime)}) [#{result_size.to_i}]"
            @logger.debug format_log_entry(name, sql.squeeze(' '))
          end
        end
      end
    end
  end

Would this work as a plugin? As a patch to Rails itself? Or did somebody else already implement a cross-platform slow query logger?

Update 2: seems like only one or two people knew about what can_search does :) I hope we’re all a little better educated.

Update: yes, I’m using these named scopes throughout the app in other places – they aren’t used only in this one controller.

Often you have an index action where you want to sort records, filter by a parameter, and maybe join on some other tables to get a result. Let’s say you’re looking at a videos controller (where videos are acts_as_taggable) and you want to filter by user_id, filter by tag name, order by video title, or rating. Maybe later, you’ll add a roles (hm:t) association and need to only show videos viewable by a certain user. How complex!

To solve this, we’re going to play with some things you may know, and finish up with a bam! pow! that’ll take your breath away.

Rather than build up some form of frankenquery with all sorts of conditionals and cases, joins, and other messing about, let’s use a brand-new bleeding edge feature of Rails: named scopes.

First, build up individual named scopes for each axis on which you wish to filter. Make sure and put the table name in that query.

    named_scope :by_user, lambda { |user_id| 
      { :conditions => ['videos.user_id = ?', user_id] }
    }

    named_scope :tag_name, lambda { |tag_name|
      { :joins => { :taggable => :tag },
      { :conditions => ['tags.name = ?', tag] }
    }

    named_scope :rating, lambda { |rating| 
      { :conditions => ['ratings_count > ?', rating] }
    }

OK, I cheated on the last one, but let’s assume you have a counter_cache on ratings count.

Now, if you have more than one scope with joins in it, you’ll need to apply this patch to your rails installation, or upgrade past 2.1.1. This will allow you to have as many joins as you like in your scopes.

Now, here’s where the magic happens: in the controller. Big shout out to protocool for this method. Let’s build up a set of all the possible scopes that we might want to use, in an array form like [ named_scope, argument ]

def index
  scopes = []
  scopes << [ :by_user, params[:user_id] ] if params[:user_id]
  scopes << [ :tag_name, params[:tag_name] ] if params[:tag_name]
  scopes << [ :rating, params[:rating] ] if params[:rating]
end

Easy, right? Very readable.

How about some ordering?

  order = { 'name' : 'videos.name ASC' }[params[:order]] || 'videos.id DESC'

Now, as you know, you can chain named scopes. So you could say Video.by_user(2).tag_name('monkeys') Let's take advantage of this, building up a chain of scopes dynamically using 'inject', starting from Video, and adding each scope we added to the array above. This is really fun magic, because it doesn't run any of the queries until the whole thing is built. I don't even know how this works, but it does. Swimmingly.

  @videos = scopes.inject(Video) {|m,v| m.scopes[v[0]].call(m, v[1]) }.paginate(:all, :order => order)

The final method looks like this:

def index
  scopes = []
  scopes << [ :by_user, params[:user_id] ] if params[:user_id]
  scopes << [ :tag_name, params[:tag_name] ] if params[:tag_name]
  scopes << [ :rating, params[:rating] ] if params[:rating]

  order = { 'name' : 'videos.name ASC' }[params[:order]] || 'videos.id DESC'

  @videos = scopes.inject(Video) {|m,v| m.scopes[v[0]].call(m, v[1]) }.paginate(:all, :order => order, :page => params[:page])
end

One final caveat. Sometimes :joins doesn’t know where to get the video id from, so if you’re using id in your app, you’ll need a slight workaround involving manually getting the pagination count, and forcing :select => ‘distinct videos.*’ in the paginate call.

If this works for you, it’s really easy to add new filtering, ordering, or even scoping to your query. For example, you can add some form of role hackery to your video

    named_scope :viewable_by, lambda { |user| 
      { :joins => { :permissions => :roles },
        :conditions => [ "roles.user_id = ? AND permissions.role = ?", user.id, "view"
    }

Controller, you replace the first scope definition with this

scopes = [ :viewable_by, current_user ]

Or, you modify the scope inject statement

    @videos = scopes.inject(Video.viewable_by(current_user)) { |m,v| ... }

If you consider this a giant hack, you’re probably at least partly right. However, the alternative in building up a complex query with many possible moving parts is just hideous. And consider this: you can unit test each part of the query on its own, in the model specs.

The default Rails sanitize helper is actually quite powerful. You can see some of its usage here:

<%= sanitize @article.body, :tags => %w(table tr td), :attributes => %w(id class style) %>

However, as the docs say,

Please note that sanitizing user-provided text does not 
guarantee that the resulting markup is valid.

We were having an issue with users providing bad markup and leaving their tags unclosed.

This is <a href="http://foo.com">my dog<a/> and he&#8217;s super cool!

We solved it by running Hpricot over their input.

before_save :clean_html
def clean_html
  self.body = Hpricot(body).to_html
end

For performance reasons, you should probably run the hpricot and sanitize methods on the way into the database, rather than rendering it in the views, because it’s somewhat slow, and is a calculation that you only need to perform once.

In fact, instead of saving it in a callback, you could overload the accessor like so:

def body=(new_body)
  write_attribute :body, Hpricot(new_body).to_html
end

You’ll want to include the ActionView methods from ActionView::Helpers::SanitizeHelper to get ‘sanitize’ available in your model.

When you have multiple Rails applications that don’t share a common database and you want to share the user authentication information – or rather, use one app to provide authentication for another – there are a few options. Here’s how I solved it recently. This is the simplest way I could think of to get this working. I couldn’t find a plugin to do this, so here’s the result of my pdi.

Effectively what we’re doing is separating the user’s data - their profile info, if you like - from the credentials, and moving the latter to ActiveResource. This is something you should do in your own apps. Too frequently we stuff a bunch of data (like full name, phone number) into the user model, because it’s there. A more advanced version of this code might use the ‘profile’ as the resource name, updating the local profile with data from remote, and keeping User as a pure credential model.

Let’s assume we have App A which will act as the authenticator master. Our other application, App B, will still hold a User record, but we’ll override the authenticate method to use ActiveResource. We’ll also store some other fields like username and email, and will grab those each time the user logs in. That way, they can set an auth token in App A and they can login from cookies in app B (provided the cookie domain is shared).

class User < ActiveRecord::Base

  class Auth < ActiveResource::Base
    self.site = "http://app-a.com"
    self.format = :json
    self.element_name = 'user' # this is the name of the resource in your app
  end

  def self.authenticate(login, password)
    Auth.user = login
    Auth.password = password

    # Authenticating against the app will actually 'prove' the login/pass details.
    # We also want the user's details so we can cache them here.
    authed = Auth.find :first, :params => { :login => login }
    return false unless authed

    # Now, pull the data from remote and store it locally.
    user = User.find_or_initialize_by_login(login)
    user.attributes = authed.attributes
    user.save!
    user.activate!
    user

  rescue ActiveResource::ClientError # 406 error -- bad username/password.
    false
  end

Interestingly enough, find first actually runs the ‘index’ action, and returns the first record. sigh

Now, in your App A: users_controller, you want to set up a filter in the index like so:

  def index
    if params[:login]
      # for single-sign-on.
      @users = User.find(:all, :conditions => { :login => params[:login] })
    else
      @users = User.paginate(:all, :page => params[:page]) #... 
    end

    respond_to do |format|
      format.html
      format.json { render :json => @users }
    end
  end

Do you have a better way of doing this?

I’ve been trying to hire this badass designer since I first saw his work, when he was still in college and I didn’t actually have a company or the means to hire him. Anyway, years later, and I’m delighted to announce that Kyle has joined our roster as a designer and UI expert.

We’re already working his little fingers to the bone on some of our upcoming applications. Duck!

Update your rubygems to 1.1.0

Lots of really cool fixes and features are going into Rails right now. I’m sure you all subscribe to Ryan Daigle’s “What’s new in Edge Rails” at http://ryandaigle.com/—if you don’t, go do it right now.

One of the advantages of having a Rails core committer on the team at ENTP is that we get to drink from the firehose of pain that is edge rails. Many a Campfire discussion ends like this: “There’s this new feature which will solve this problem”, says Rick, “but we’ll have to move to Edge”.

Anyway, it looks like 2.1 is on the horizon, which means lots of updates as they get to pushing it out the door. Here are some of the awesome new features:

Timestamped migrations

Migrations will soon have a UTC timestamp instead of a number, so it’ll be much easier to merge in branches and fix your migrations without errors.

named_scope

named_scope is a way of providing association-like scopes to your models, so you can define things like

named_scope :visible, :conditions => { :visible => true }

User.visible.find :first

which, in my opinion, is much better than with_scope, because it helps you to look for a method called ‘visible’ rather than “find_visible”, which is some nasty meta guff.

Dirty field checking

You’ll be able to check, and commit, changes to single fields, rather than updating entire rows. This is still shiny-new, and I haven’t had a chance to use it yet, but here’s a sample from the feature’s tests:

person.name = 'a'
assert person.name_changed?

config.gem

You can specify which gems are required by your application in the “config” section. The extra bonus with this is the new rake task, rake gems:install which will automagically install the required gems for you. Seriously, this is freakin’ awesome.

Ripped from the CHANGELOG:

Rails::Initializer.run do |config| 
   config.gems "bj" 
   config.gems "hpricot", :version => '0.6', :source => "http://code.whytheluckystiff.net" 
   config.gems "aws-s3", :lib => "aws/s3" 
 end        

rake gems # lists gems
rake gems:install  # Install all required gems: 
rake gems:unpack GEM=bj # Unpack specified gem to vendor/gems/gem_name-x.x.x

Gems can be plugins

If your plugin has the file, rails/init.rb, it will now work cleanly as a plugin.

OpenID plugin updated

The OpenID auth plugin now uses version 2 of the gem, which is a major leap forward.

Massive documentation patch

Pratik’s rails-documentation github project is certainly a lively one; look forward to some Doc Project updates over the coming weeks on this front. Now that Rails core has some more, dedicated doc-loving committers, Rails docs will be progressing at a major pace.

Phew. What an update! The upgrade from 2.0.2 to 2.1 will certainly be an interesting one for us all.

Oh, and Ryan? Scoop!

Today I'm going to show you how I combine my old-school Javascript with the latest Prototype has to offer. I hope you learn a little about the basic use and misuse of "bind". Hopefully someone will chime into the comments and give a better way to do this code!

You may know that you can add a global ajax responder to your application:

Ajax.Responders.register({
  onCreate: function() {
    Ajax.activeRequestCount++;
  },
  onComplete: function() {
    Ajax.activeRequestCount--;
  }
});

But I'm not going to cover that here, because you can't get the originating object. For example, there's no way here for a global ajax responder to get the anchor object.

<a class="add" href="#" onclick="new Ajax.Request('/products/3/categorizations/18', {asynchronous:true, evalScripts:true, method:'put'}); return false;">add</a>

Here's the rails generator code:

<%= link_to_remote "add", { :url => product_categorization_path(@product, category), :method => :put } %>

So, let's add some code to prevent the user from clicking twice.
To keep the view clean, I'll implement it as a simple view helper in application_helper.rb

<%= link_to_remote "add", { :url => product_categorization_path(@product, category), :before => ajax_progress, :method => :put } %>

def ajax_progress
  "setTimeout(function(){ this.innerHTML = '...' }.bind(this), 100)"
end

SetTimeout takes either a function name or an anonymous function. I'm binding the anonymous function to "this", which in this context refers to the 'A' anchor tag. Because it's bound, I can refer to "this" inside the function and get the anchor tag.

The setTimeout is useful because it lets us modify the tag (even remove it from the DOM) without messing with the ajax request. You can use setTimeout on a form to change the action, so it can't be submitted twice (for important forms)

I'm going to step it up one more, because this code belongs in a library. Time to open up application.js and create a ghetto pseudoclass singleton thingy.

MyApp = {
  /* do this while we're waiting for ajax responses */
  ajaxing: function(){
    this.innerHTML = '...'; 
    this.onclick = FacetApp.nothing;
  },

  /* don't do anything. useful for descriptive declarations like $('a').onclick = FacetApp.nothing */
  nothing: function() { 
    return false 
  }
}

This will prevent the link from doing anything if the user clicks it twice.

Now, to modify the ajax_progress helper to simple beauty.

def ajax_progress
  "setTimeout(MyApp.ajaxing.bind(this), 100)"
end

Finally, let's show the user a GMail-style notice after 5 seconds, just to let them know that we're running slow or have just plain died.

MyApp = {

  ajaxing: function(){
    this.innerHTML = '...'; 
    this.onclick = FacetApp.nothing;
    setTimeout(MyApp.slooow.bind(this), 5000);
  },

  /* shows a ? symbol. useful for showing progress on an ajax action */
  slooow: function(){
    this.innerHTML = ".?."
    /* show a warning message in the ui somewhere */
  }
}

Got a better way of doing this?

This happened a few weeks ago but now that he’s settled in, I’d like to welcome Mr Neighborly to the ENTP fold. We totally stole him from Chow, (sorry, Evan).

Readers of this blog will be familiar with his efforts in the Rails community. He’s also a damn fine coder. As well as your standard Rails application hacking, Jeremy will also be working with us on building both training materials, and the toolchain to write them. Now that he’s in-house, we can re-start some of the projects that have been on the backburner.

Jeremy’s currently spearheading efforts on the new Ruby on Rails official wiki (shh!). We’ll be re-launching it on a new platform with spam protection and lots of gardening. Combined with the Github documentation repository, there are many exciting developments in the Rails documentation world. We’ll be backing up this burst of activity with some innovative docproject-backed funding in the coming weeks.

Following the previous article, I took the existing "acts_as_flaggable" plugin, extended it, rewrote most of it, added some generators and test harness from Acts as Authenticated, and voila -- an experimental plugin.

I've gone a similar route to actsas_authenticated and attachmentfu, in that some code is in the plugin, and a chunk of it is generated and installed into your application. I'm considering an alternate way of generating enough code that you can delete the plugin (or reinstall later). Anyway..

If you're on git, you can clone the source from

git clone git://github.com/courtenay/can_flag.git

and symlink it to your vendor/plugins directory.

Or, download as a tarball from

http://github.com/courtenay/can_flag/tarball/master

and pop it in your vendor/plugins directory.

What does it do?

First, you'll need to follow the directions in the README.

script/generate can_flag flags

This will create 'flags_controller' and associated files, route and views. It will also create a "_flag" partial in app/views/layouts, which you can modify and include throughout your application like

<%= render :partial => "layouts/flag", :object => @topic %>

You'll then need to set up your user model

class User < ActiveRecord::Base
  can_flag
end

and set up your article, topic, or whatever other content model you want to be flaggable. You can set this in as many classes as you like. There is also an "after_flagged" callback that you can use in the content model.

class Content < ActiveRecord::Base
  can_be_flagged :reasons => [ :spam, :troll, :inappropriate ]

  # optional callback
  def after_flagged
    # do something cool
    # censor! if flags.size > 5
    # user.suspend! if user.flaggings.size > 5 
  end
end

Of course the actions you perform in the 'after_flagged' callback are entirely optional and application-specific. The "Flag" model lives in the plugin directory. If you want to modify it, you can copy it into app/models.

You'll have a few new associations.

content has_many :flags
user has_many :flags # they reported
user has_many :flagging # they created this filth

Have a play with this very quickly produced code and let me know how you go.

On my todo list:

• generate specs for the generated code
• generate tests for those old-school test/unit peeps
• write a few more tests for the actual generation and models.
• add in user expert rankings as an option, so that good censors (aka prudes) can have more power
• better feedback on flags, like "good" or "bad", rather than just "delete" as a way of clearing
• add in positive as well as negative (let's not overlap with acts_as_rated)