Premcaching, updated

Courtenay : October 10th, 2007

See my previous article on premcaching, preloading data and stuffing it into memcached in a fork.

Paul McKellar just sent this snippet to re-establish the database connection in your fork.

def fork_with_new_connection(config, klass = ActiveRecord::Base, &block)
  fork do
    begin
      klass.establish_connection(config)
      yield
    ensure
      klass.remove_connection
    end
  end
end
def fire_and_forget(&block)
  config = ActiveRecord::Base.remove_connection
  pid = fork_with_new_connection(config) do
    begin
      yield
    ensure
      Process.exit!
    end
  end
  ActiveRecord::Base.establish_connection(config)
  Process.detach pid
end

Awesome. Is anyone else using techniques like this for their crazy scaling or pagination needs?

Posted by Courtenay Filed in rails, ruby

5 Responses to “Premcaching, updated”

  1. Per Wigren aka Tuxie Says:
    October 11th, 2007 at 12:59 AM

    We use a technique similar to the fireandforget method above, but without the AR connection stuff. Basically (simplified):

    def fork_off
    return yield if RAILS_ENV == 'test'
    pid = fork do
        begin
            yield
        ensure
            exit!
        end
    end
    Process.detach(pid)
end

    It works great for simple background stuff as long as you don't reuse established sockets. We use AR to fetch the data we need into an array and then use the array inside the fork_off block.

    If an answer/result is needed it can HTTP POST the result back to the application, using ActiveResource, net/http or something else.

  2. Per Wigren aka Tuxie Says:
    October 11th, 2007 at 01:15 AM

    Heh, I just saw that you had nearly identical code in the previous article. Sorry. I should had read the original article before posting. :)

  3. Per Wigren aka Tuxie Says:
    October 11th, 2007 at 01:55 PM

    The timing is almost scary... Just today we noticed some problems using this method. Because the fork still has the http socket open, haproxy won't reuse the mongrel instance until both the original request and the fork has finished.

    The solution is to kill the socket inside the fork. As a bonus, I also make ActiveRecord forget about its connections so it will re-connect upon use:

    def fork_off
  return yield if RAILS_ENV == 'test'
  pid = fork do
    begin
      ActiveRecord::Base.active_connections.delete_if{true}
      request.cgi.instance_eval('@response').socket.close rescue nil
      yield
    ensure
      exit!
    end
  end
  Process.detach(pid)
  pid
end
  4. Per Wigren aka Tuxie Says:
    October 12th, 2007 at 02:05 AM

    Sorry for spamming. :) This is a better way to close the sockets than the version above, which is Mongrel-specific:

    ObjectSpace.each_object(BasicSocket) { |s| s.close rescue nil }
  5. Buzz Says:
    October 14th, 2007 at 08:51 PM

    I agree with Per about closing the sockets... But the code is really cool..

    Thanks for sharing.

Sorry, comments are closed for this article.