Sometimes you find yourself with eighty controllers and it becomes too large to fit in your head.

In this case, you may want to think about separating parts of your mega application into smaller applications. This will help your team work on different parts in parallel.

Unfortunately, routing doesn’t currently support this. While in most cases you’re only linking to index or show from another controller (you don’t really PUT or POST from another controller ) it still requires some kind of helper methods (def users_path ; “http://”.. ) or routing hack.

Jake Howerton has just published a plugin that hacks routes to solve just this problem: Multi App Routing Check it out. The module_eval stuff scares me, but then again, I just don’t want to understand the mechanics of routing.

Config is done with a yaml file, and the syntax looks like

map.application(:billing, :protocol => ‘https’) do |billing|
  billing.resources :payments
end

Nice and simple.

Taking a vanilla rails application from one box and up is a fun process. The exact path you’ll take depends on the nature of your data, and the ratio of database reads to writes. I’m going to cover some of the more common use cases. If you don’t want to get your hands dirty and it’s kind of an emergency, look at stage zero, then skip to the end where I tell you who you can just pay to fix it.

The path you’ll take also depends on how much money you have to play with, and how quickly your site is growing. For example, if you’re sitting on a mountain of cash, and the facebook users are coming in like lemmings, then you can just throw hardware at it. However, if things are tight, and it’s a nice linear growth curve, then you can play around with caching.

Let’s assume you have a slice or VPS (if you’re on shared hosting, the first step should be to get a dedicated box or at least a xen instance).

Stage zero: fix any ‘duh’ errors

Make sure you’re on a database that can handle the load. This doesn’t include sqlite. I’m going to suggest MySQL in this article, because it’s where I have the most experience.

Make sure you’re not serving up static files through mongrel. This will happen if you are proxying everything through the webserver.

Upgrade your webserver to something like nginx. Alternatively, you might use pound as a load balancer, pointing dynamic requests at mongrels, and static requests at lighttpd. (Interested? I can write an article on this. Let me know.)

Move off that $20/month shared box and get your own server. You can lease a phat server in a data center on 100mbit pipe for $100/month. If you want to colo, I recommend Corporate Colo in Los Angeles.

Move any slow actions into a dedicated process. For example, you have some code that takes 4 seconds to update a bunch of tables? You probably want to fire an asynchronous event to a BackgrounDRB process that handles this exclusively.

Move your uploads to a dedicated merb cluster – it’s like a cut-down rails with less magic and more speed.

Stage one: clean up your database

Take a look at your logs – are you performing over 10 database calls per request? You need to fix this. Are you performing over 90? You’re a dumbass. (yes, even I am guilty of this).

Generally you can reduce the number of requests by denormalizing; for example, you have a list of users and a count of how many comments they’ve made.

<% @users.each do |user| %>
  <%=h user.name %> (<%= user.comments.count %>)
<% end %>

You’re performing a “COUNT” for every single user, every time the page loads. Yuk! This is a “read-optimizable” situation, since there are many READS for each WRITE (comments don’t get created that often).

Add a counter_cache to the comments belongs_to :user association and change this to

<%=h user.name %> (<%= user.comments_count %>)

You can do this in other situations where you’re chaining through associations.

Your task is <%=h @list_item.task.name %>.

This call has to find the list_item task, and then grab the name. You can fix this by either adding :include to the ListItem.find, or, you can denormalize the task name.

Include isn’t always an option, and can be slow. Nothing’s faster than denormalizing. Add a “taskname” to the listitem model.

Your task is <%=h @list_item.task_name %>.

Then make sure to update that field if the task gets updated.

class Task < ActiveRecord::Base
  has_many :list_items
  after_save :update_list_item_names

  def update_list_item_names
    list_items.each { |li| li.update_attribute(:name, self.name) }
  end
end

Yes, there are faster ways of doing this, and yes, I should probably wrap that in a transaction. But you get the point. (note to self: if rails had dirty-field checking, this would be much better)

Stage two: cache the hell out of it

Next thing you want to do: caching. If you haven't already, install memcached, use the cache-fu plugin, and start saving the results of long-running or frequent queries into the cache. Set the TTL (timeout) at about 5 minutes; that way you won't need to write any expiry code (it's lazy, but you're busy!) You'll immediately notice a drop in load. If you have time, write some cache-expiring observers and up the TTL to 15 minutes or even an hour.

Eventually you want to have memcached sitting between your application and the database. Most of your database calls' results will be stored for at least 5 minutes, and maybe forever, in memcached.

If you can, add some action caching. Action cache is like page caching, but it runs any filters you may have. Action caching isn't always easy, particularly if you have "current_user" dependent code in your views. I have a solution for this which I'll be releasing soon, but in the mean time, you may not be able to action cache. Any action-cached pages will be vastly beneficial to your load, and combining memcached with action caches means that you can virtually eliminate any database slowness and is almost as good as the page cache.

If you can action cache, then you can probably page cache. A page-cached site will get you about 3,000 requests per second, thereabouts, and a simple GET request won't even hit your application; you're serving raw html through the webserver. You will soon start thinking of rails as an HTML generator, rather than an app server.

However, all these caching measures won't hide a basic problem: you are performing lots of database queries, and it's harshing your mellow.

Stage three: move the database to another server

This should be fairly painless. Get yourself a fat database server. By fat I mean, super-fast disks, plenty of RAM, and the fastest networking you can afford.

Set it up so that it's only accessible from your main box, which will now be known as the app server. Point your database.yml at the IP of the database server.

Now your app server has much less load, so you can increase the number of mongrels. Add some more RAM to the app server box, too, if you can.

How many mongrels?

Here's a simple formula to follow.

A. Take the (average or median) request time, in seconds.  Say, 0.250 seconds (250ms)
B.  How many requests do you want to handle at peak?  (e.g. 10,000 a minute, 166 a second)
C.  Multiply A x B :  0.250 * 166 = 41.5

So you need about 40 mongrels to handle the load. At about 60MB per listener, that's 2.4GB of RAM, plus a bit of room for leakiness and swapping. Ezra at Engine Yard suggests "about 10 dogs per CPU core", which means that if we have a 4-core opteron box with 3GB of RAM, then this is possible on one box.

Your mileage will vary, which means, if the box is lagging, remove a few mongrels.

Stage four: add more servers as necessary

Here's where it gets interesting. Which of your servers has the most load?

If it's the app server, then setup another box as an exact copy. Now you have app1 and app2. You will need to load balance between app1 and app2. You can do this with a hardware load balancer, or you can use pound on app1 to balance to listeners on app2. (You'll have a single point of failure on app1 if you do it this way)

If the db server is the most heavily loaded box, things start getting interesting: you'll either need some kind of replication, or you'll need to shard (partition) your data.

Replication vs Sharding

Take a look at the data in your application. If it were a person, would it be "extroverted" or "introverted"? That is, could you split the data into many sections (no friends, introverted), or is it all cross-linked (lots of friends)?

For example, you are hosting subversion repositories. You can easily send half the records to one database and half to the other. Or, you host thousands of social networks, each with about 50 users (collectivex, I’m looking at you.)

In this case, one database box would handle all users with names A–L, and another box from M–Z.

If you have a social networking site where anyone can be friends with anyone else, you’re going to have difficulty partitioning the data. (Astute readers will instantly think about denormalizing to make this still possible).

If you have one shared table (users) but the rest of the data can be sharded, then you will want some bastard stepchild method.

Replication: Master-Slave

So, replication (MySQL only from here on). Let’s say you have a few writes (inserts, updates) and a lot of SELECTs. Most people are just viewing things, not updating records. This is fun and easy.

You set up one database box as the “master”. This box will behave as normal. You can read and write data as before.

You then set up as many “slave” boxes as you like. These boxes will be read-only, but because you have a large amount of reads, then much of the load can be pushed out to these slaves. You’ll need to hack at your rails app to direct simple reads at a slave DB. Luckily, someone’s already done the work and called it acts_as_readonlyable.

The problem here is that the slaves will always be lagging, depending on load. Under light load, they may only be 100ms behind. Under heavy load, you can’t be guaranteed of any sort of synchronization. In this case, you’ll want to use memcached heavily. Here’s some cache-fu code.

class Category < ActiveRecord::Base
  acts_as_cached

  def after_save
    Category.set_cache(id, self)
  end
end

When you save the category, it pushes the record (self) into memcached. That means, with a long TTL, you'll never need to do a simple “find” on category from the database, and replication lag won’t matter.

Finally, you’ll want to load-balance to the database servers, an exercise left to the reader.

      / write==[db1] master
[app1]
     \                /==[db2] slave
      \ read==LB==[db3] slave
                       \==[db4] slave

Replication: Simple Master-Master

In this situation, you have two sets of stacks. Each stack has an app box and a database box. They are almost identical; the app server is wired to one database server. There is no crossover. ASCII-tabulous diagram:

   /[app1]====[db1]  master+slave
LB                        |    replicate
   [app2]====[db2]  master+slave

Both databases in this case are masters. That means, both act as masters, but both act as slaves as well. You can even set up the boxes so that if one goes down, the other fails over and takes on both IP addresses.

Because the setup is asynchronous, you need to assign each database a separate set of autoincrement keys. DB1 will increment values like 11, 21, 31, 41, 51, and DB2 ids will increment like 12, 22, 32, 42, 52. You set these with auto_increment_increment and auto_increment_offset.

Take care! If you have a UNIQUE index on other, non-auto-increment fields, you need to make sure that the same database will be used for CREATEs. You’ll need some algorithm, such as checking the final character or number of the unique field. You’ll also need some way of redirecting writes to a specific database, as well as dealing with load balancers. You may find MySQL Proxy useful here – you can use Lua to control the load-balancing at the db layer.

Master:Master replication doesn’t really scale past 10 boxes, because the databases will be so busy updating that they won’t be able to serve requests. However, 99% of rails applications won’t get to this stage.

And remember – there will be some replication lag between the boxes, so your code will need to be tolerant of this issue.

Stage Six: More boxes!

At this stage, you should have most of your data stored in memcached, and it’s time to get yourself a dedicated memcached box with gigabit networking and a metric crapton of RAM.

Your data should be nicely segmented (sharded, or partitioned) into separate databases.

Stage Seven: You’re Going To Need Help

If you’ve roughly followed all of the above steps, and your site is still lagging, you either didn’t follow the instructions, or you’re beyond this and need to bring in some experts. Replicating and sharding should cover most people’s scaling needs, such that you just keep adding stacks of app+db and expanding the memcached cluster.

You can hire skilled rails consultants to clean up your code (there are plenty of #caboosers with the requisite experience), or you can use a hosting service like Engine Yard (staffed almost exclusively with caboosers) where they will have your app running on a cluster pretty much like I’ve described above, only bigger and faster. It’s going to cost you, but you get what you pay for. Hell, they even deploy your app for you.

Developers, if you’re not an avid reader of the MySQL Performance Blog, go subscribe now. They are also for hire, and I hear they’re most excellent. Pricey, but worth every damn penny.

Got any more tricks? Hook me up in the comments.

My independent migrations patch has been sitting on the rails trac for 4 months, so I’m releasing it as a plugin.

Code plz thx

ticket: http://dev.rubyonrails.org/ticket/6799
plugin: svn://caboo.se/plugins/court3nay/independent_migrations

I plan to add a generator, one day, if anyone cares.

wtf

In simple terms, here’s what it does. To paraphrase zenspider, if two migrations have the same number, Rails will bitch at you. This plugin removes the bitching. To get it working, install, then edit your migrations so they look like this

class AddProject < ActiveRecord::IndependentMigration

That’s it. Now your migration directory can look something like

002_add_project.rb
002.add_monkeys.rb
003.fix_monkeys.rb

and as long as all 002 are independent, then the migration will run. This patch is only guaranteed for Rails 1.2.3 and may change as ActiveRecord changes as it relies heavily on monkeypatching (or reopening classes, as David Black would have you believe) some private methods.

Why?

If you’re still wondering, “Why?”, the usage came from heavy usage of branches. Basically, if there are two branches being developed in parallel, and both are getting migrations, it can be assumed that those migrations are, in fact, in parallel, and won’t conflict, so can have the same number.

Alternate solutions

There is of course more ways to skin a cat, and the other contender for your migrations is Françoise Beausoleil’s “Timestamped Migrations”, while it isn’t a plugin (yet), numbers your migrations like it says on the box (with timestamps).

http://blog.teksol.info/articles/search?q=migration

Capistrano memcached tasks

Capistrano rocks for deployment. If you haven’t ever used it to deploy an app, you’re seriously missing out. There’s a lot of good recipes floating around online and this stuff changes so often that there’s probably something out there that’s better. However I figured I’d share how I’m currently managing our memcache daemons at work.

memcached, rails wtf?

We’re using defunkt’s cache_fu plugin right now to cache models in our system. It’s pretty straightforward to use; an acts_as class method, and a config/memcached.yml file. The memcached.yml file lets you provide one or more servers that the memcache clients can connect to. Unfortunately there’s no centralized way to manage those memcache daemons on your deploy hosts. With our good friend cap, some ruby, and a little time invested we can roll our own solution.

Whatcha Want?

The requirements for this task are dead simple.

• use cap to get out to the remote hosts
• the memcache daemon list comes from cache_fu’s yml file
• write a script to start, stop, restart, get the status of, and commit mass genocide against(I want all processes kill -9’d right now).

It turned out to be a pretty simple hack; parse the yaml, get the host’s ip address, see if the host is in the server list, if it is do whatever command line operation we were given. Check it out.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

#!/usr/bin/env ruby
# this goes in your script/ directory
# it parses your memcached.yml file and hooks you up w/ some info
# it keeps you from having to mess w/ stale memcached daemons for whatever reason.
require 'yaml'

class MemcachedCtl
  attr_accessor :memcached, :memory, :pids, :servers, :ip_address, :ethernet_device
  def initialize
    env = ENV['RAILS_ENV'] || 'development'
    self.memcached = `which memcached`.chomp
    self.servers = [ ]
    self.pids    = { }
    self.ethernet_device = ENV['ETH'] || 'eth0'
    self.ip_address = get_ip_address || '0.0.0.0'
    self.memory = '128'
    
    config = YAML.load_file(File.expand_path(File.dirname(__FILE__) + "/../config/memcached.yml"))
    self.servers = [ config['defaults']['servers'] ].flatten rescue ['127.0.0.1:11211']
    self.servers = [ config[env]['servers'] ].flatten if config[env]['servers']
    self.servers.reject! { |server| host,port = server.split(/:/); self.ip_address == host }
    self.memory = config[env]['memory'] unless config[env]['memory'].nil?
    
    each_server do |host,port|
      `ps auwwx | grep memcached | grep '\\-l #{ip_address} \\-p #{port}' | grep -v grep`.split(/\n/).each do |line|
        self.pids[port] = line.split(/\s+/)[1]
      end
      self.pids[port] ||= 'Down'
    end
  end
  
  def execute(cmd)
    send(cmd)
  end
  
  def restart; stop; sleep 1; start; end
  
  def status
    each_server { |host,port| puts "Port #{port} -> #{pids[port] =~ /\d+/ ? 'Up' : 'Down'}" }
  end
  
  def kill
    each_server { |host,port| `kill -9 #{pids[port]} > /dev/null 2>&1` if pids[port] =~ /\d+/ }
  end
  
  def stop; kill; end
  
  def start
    each_server do |host,port|
      `#{memcached} -d -m #{memory} -l #{ip_address} -p #{port}`
      STDERR.puts "Try memcached_ctl status" unless $? == 0
    end
  end
    
  protected
  def each_server(&block)
    servers.each do |server|
      host,port = server.split(/:/)
      yield host, port
    end
  end
  
  def get_ip_address # this works on linux you might have to tweak this on other oses
    line = `/sbin/ifconfig #{ethernet_device} | grep inet | grep -v inet6`.chomp
    if line =~ /\s*inet addr:((\d+\.){3}\d+)\s+.*/
      self.ip_address = $1
    end
  end
end
###########################################################################
cmd = ARGV.shift
unless cmd.nil?
  MemcachedCtl.new.execute(cmd)
end

Get Your Cap On

So I named the script memcached_ctl and added it to my script/ directory in svn and redeployed. Now I can start poppin caps in our memcache daemons. If I throw the following code at the bottom of my config/deploy.rb I can see what’s up with my memcache daemons on my production hosts. If I were smart/motivated I’d make a gem and let you good people include these tasks, you know like you should be doing for mongrel_cluster.

1
2
3
4
5
6
7
8
9

# ====================================
# Memcached Server TASKS
# ====================================
%w(start stop restart kill status).each do |cmd|
  desc "#{cmd} your memcached servers"
  task "memcached_#{cmd}".to_sym, :roles => :app do
    run "RAILS_ENV=production #{ruby} #{current_path}/script/memcached_ctl #{cmd}"
  end
end

Checkout how easy it is to work with my memcache daemons now.

mpro% cap memcached_status
  * executing task memcached_status
  * executing "RAILS_ENV=production /opt/local/bin/ruby /home/production/Sites/oracle/current/script/memcached_ctl status" 
    servers: ["10.2.1.29", "10.2.1.30"]
    [10.2.1.29] executing command
    [10.2.1.30] executing command
 ** [out :: 10.2.1.29] Port 11212 -> Up
 ** [out :: 10.2.1.30] Port 11212 -> Up
    command finished
mpro% cap memcached_stop  
  * executing task memcached_stop
  * executing "RAILS_ENV=production /opt/local/bin/ruby /home/production/Sites/oracle/current/script/memcached_ctl stop" 
    servers: ["10.2.1.29", "10.2.1.30"]
    [10.2.1.29] executing command
    [10.2.1.30] executing command
    command finished
mpro% cap memcached_status
  * executing task memcached_status
  * executing "RAILS_ENV=production /opt/local/bin/ruby /home/production/Sites/oracle/current/script/memcached_ctl status" 
    servers: ["10.2.1.29", "10.2.1.30"]
    [10.2.1.29] executing command
    [10.2.1.30] executing command
 ** [out :: 10.2.1.29] Port 11212 -> Down
 ** [out :: 10.2.1.30] Port 11212 -> Down
    command finished
mpro% cap memcached_start
  * executing task memcached_start
  * executing "RAILS_ENV=production /opt/local/bin/ruby /home/production/Sites/oracle/current/script/memcached_ctl start" 
    servers: ["10.2.1.29", "10.2.1.30"]
    [10.2.1.29] executing command
    [10.2.1.30] executing command
    command finished
mpro% cap memcached_status
  * executing task memcached_status
  * executing "RAILS_ENV=production /opt/local/bin/ruby /home/production/Sites/oracle/current/script/memcached_ctl status" 
    servers: ["10.2.1.29", "10.2.1.30"]
    [10.2.1.29] executing command
    [10.2.1.30] executing command
 ** [out :: 10.2.1.29] Port 11212 -> Up
 ** [out :: 10.2.1.30] Port 11212 -> Up
    command finished

bust your own caps.

I really dig this approach because I can deploy all sorts of apps without ever logging in and starting things manually. Things that you bounce less(like your memcacheds) are even bigger candidates for automation. At any given time I can tell you whether they’re up or down, and I can restart them in a few seconds if you so desire. This kind of approach, while really simple, works really well with Capistrano. So get out there are start bustin caps.