Capistrano memcached tasks

Capistrano rocks for deployment. If you haven’t ever used it to deploy an app, you’re seriously missing out. There’s a lot of good recipes floating around online and this stuff changes so often that there’s probably something out there that’s better. However I figured I’d share how I’m currently managing our memcache daemons at work.

memcached, rails wtf?

We’re using defunkt’s cache_fu plugin right now to cache models in our system. It’s pretty straightforward to use; an acts_as class method, and a config/memcached.yml file. The memcached.yml file lets you provide one or more servers that the memcache clients can connect to. Unfortunately there’s no centralized way to manage those memcache daemons on your deploy hosts. With our good friend cap, some ruby, and a little time invested we can roll our own solution.

Whatcha Want?

The requirements for this task are dead simple.

• use cap to get out to the remote hosts
• the memcache daemon list comes from cache_fu’s yml file
• write a script to start, stop, restart, get the status of, and commit mass genocide against(I want all processes kill -9’d right now).

It turned out to be a pretty simple hack; parse the yaml, get the host’s ip address, see if the host is in the server list, if it is do whatever command line operation we were given. Check it out.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

#!/usr/bin/env ruby
# this goes in your script/ directory
# it parses your memcached.yml file and hooks you up w/ some info
# it keeps you from having to mess w/ stale memcached daemons for whatever reason.
require 'yaml'

class MemcachedCtl
  attr_accessor :memcached, :memory, :pids, :servers, :ip_address, :ethernet_device
  def initialize
    env = ENV['RAILS_ENV'] || 'development'
    self.memcached = `which memcached`.chomp
    self.servers = [ ]
    self.pids    = { }
    self.ethernet_device = ENV['ETH'] || 'eth0'
    self.ip_address = get_ip_address || '0.0.0.0'
    self.memory = '128'
    
    config = YAML.load_file(File.expand_path(File.dirname(__FILE__) + "/../config/memcached.yml"))
    self.servers = [ config['defaults']['servers'] ].flatten rescue ['127.0.0.1:11211']
    self.servers = [ config[env]['servers'] ].flatten if config[env]['servers']
    self.servers.reject! { |server| host,port = server.split(/:/); self.ip_address == host }
    self.memory = config[env]['memory'] unless config[env]['memory'].nil?
    
    each_server do |host,port|
      `ps auwwx | grep memcached | grep '\\-l #{ip_address} \\-p #{port}' | grep -v grep`.split(/\n/).each do |line|
        self.pids[port] = line.split(/\s+/)[1]
      end
      self.pids[port] ||= 'Down'
    end
  end
  
  def execute(cmd)
    send(cmd)
  end
  
  def restart; stop; sleep 1; start; end
  
  def status
    each_server { |host,port| puts "Port #{port} -> #{pids[port] =~ /\d+/ ? 'Up' : 'Down'}" }
  end
  
  def kill
    each_server { |host,port| `kill -9 #{pids[port]} > /dev/null 2>&1` if pids[port] =~ /\d+/ }
  end
  
  def stop; kill; end
  
  def start
    each_server do |host,port|
      `#{memcached} -d -m #{memory} -l #{ip_address} -p #{port}`
      STDERR.puts "Try memcached_ctl status" unless $? == 0
    end
  end
    
  protected
  def each_server(&block)
    servers.each do |server|
      host,port = server.split(/:/)
      yield host, port
    end
  end
  
  def get_ip_address # this works on linux you might have to tweak this on other oses
    line = `/sbin/ifconfig #{ethernet_device} | grep inet | grep -v inet6`.chomp
    if line =~ /\s*inet addr:((\d+\.){3}\d+)\s+.*/
      self.ip_address = $1
    end
  end
end
###########################################################################
cmd = ARGV.shift
unless cmd.nil?
  MemcachedCtl.new.execute(cmd)
end

Get Your Cap On

So I named the script memcached_ctl and added it to my script/ directory in svn and redeployed. Now I can start poppin caps in our memcache daemons. If I throw the following code at the bottom of my config/deploy.rb I can see what’s up with my memcache daemons on my production hosts. If I were smart/motivated I’d make a gem and let you good people include these tasks, you know like you should be doing for mongrel_cluster.

1
2
3
4
5
6
7
8
9

# ====================================
# Memcached Server TASKS
# ====================================
%w(start stop restart kill status).each do |cmd|
  desc "#{cmd} your memcached servers"
  task "memcached_#{cmd}".to_sym, :roles => :app do
    run "RAILS_ENV=production #{ruby} #{current_path}/script/memcached_ctl #{cmd}"
  end
end

Checkout how easy it is to work with my memcache daemons now.

mpro% cap memcached_status
  * executing task memcached_status
  * executing "RAILS_ENV=production /opt/local/bin/ruby /home/production/Sites/oracle/current/script/memcached_ctl status" 
    servers: ["10.2.1.29", "10.2.1.30"]
    [10.2.1.29] executing command
    [10.2.1.30] executing command
 ** [out :: 10.2.1.29] Port 11212 -> Up
 ** [out :: 10.2.1.30] Port 11212 -> Up
    command finished
mpro% cap memcached_stop  
  * executing task memcached_stop
  * executing "RAILS_ENV=production /opt/local/bin/ruby /home/production/Sites/oracle/current/script/memcached_ctl stop" 
    servers: ["10.2.1.29", "10.2.1.30"]
    [10.2.1.29] executing command
    [10.2.1.30] executing command
    command finished
mpro% cap memcached_status
  * executing task memcached_status
  * executing "RAILS_ENV=production /opt/local/bin/ruby /home/production/Sites/oracle/current/script/memcached_ctl status" 
    servers: ["10.2.1.29", "10.2.1.30"]
    [10.2.1.29] executing command
    [10.2.1.30] executing command
 ** [out :: 10.2.1.29] Port 11212 -> Down
 ** [out :: 10.2.1.30] Port 11212 -> Down
    command finished
mpro% cap memcached_start
  * executing task memcached_start
  * executing "RAILS_ENV=production /opt/local/bin/ruby /home/production/Sites/oracle/current/script/memcached_ctl start" 
    servers: ["10.2.1.29", "10.2.1.30"]
    [10.2.1.29] executing command
    [10.2.1.30] executing command
    command finished
mpro% cap memcached_status
  * executing task memcached_status
  * executing "RAILS_ENV=production /opt/local/bin/ruby /home/production/Sites/oracle/current/script/memcached_ctl status" 
    servers: ["10.2.1.29", "10.2.1.30"]
    [10.2.1.29] executing command
    [10.2.1.30] executing command
 ** [out :: 10.2.1.29] Port 11212 -> Up
 ** [out :: 10.2.1.30] Port 11212 -> Up
    command finished

bust your own caps.

I really dig this approach because I can deploy all sorts of apps without ever logging in and starting things manually. Things that you bounce less(like your memcacheds) are even bigger candidates for automation. At any given time I can tell you whether they’re up or down, and I can restart them in a few seconds if you so desire. This kind of approach, while really simple, works really well with Capistrano. So get out there are start bustin caps.