Nepal on Rails

millisami's rants!

Setting Up Resque, Status, Retry, Multiple Failure Backend With Rails 3

Setting up Resque is easy-peasy with Rails app. But after adding couple of plugins, things gets messy and sometimes you just can’t figure out that its really working as it supposed to.

Later I added resque-status to monitor the health of the jobs. Then added resque_mailer to send emails asynchronously. Then I added resque-retry to retry the jobs themselves to retry 3 times in 60 secs of interval. After deploying to production, it was being hard to assure/verify that its actually working as it was supposed to. Then, had to setup the resque jobs to send errors to hoptoad/airb rake that occurs inside the jobs themselves.

So, after doing the back-n-forth manual testing, the following is the setup/configurations that did exactly the way the application wanted.

Following are the gems used in the app

# Gemfile
gem resque, ~> 1.19.0, :require => resque/server
gem resque_mailer
gem resque-status
gem rufus-scheduler
gem resque-retry

Configuration for the hoptoad/airbrake

# config/initializers/hoptoad.rb
HoptoadNotifier.configure do |config|
  config.api_key = "71a95c9cb6552275638f0600b7e6b65c"
  config.development_environments = [test] # Environments added here wont be sent
  config.host    = millibrake.cloudfoundry.com
  config.port    = 80
  config.secure  = config.port == 443
end

You might stumble upon what that millibrake.cloudfoundry.com is? Well its just a clone of Hoptoad at CloudFoundry. You can setup your own following this post or you can just use the hoptoad one.

This is initializer file for resque and its addons

# config/initializers/resque.rb
require resque/server
require resque/status_server
require resque/job_with_status
require resque-retry
require resque-retry/server

require resque/failure/multiple
require resque/failure/hoptoad
require resque/failure/redis

config = YAML::load(File.open("#{Rails.root}/config/redis.yml"))[Rails.env]
Resque.redis = Redis.new(:host => config[host], :port => config[port], :thread_safe => true)

# Resque Authentication protection (view at http://)
Resque::Server.use Rack::Auth::Basic do |username, password|
  username == config[username]
  password == config[password]
end

Resque::Status.expire_in = (24 * 60 * 60) # 24hrs in seconds

# Exclude sending actual emails in these environments
Resque::Mailer.excluded_environments = [:test, :cucumber, :development]

# Failure Backends Setup
Resque::Failure::Hoptoad.configure do |config|
  config.api_key = HoptoadNotifier.configuration.api_key
end

Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis, Resque::Failure::Hoptoad]
Resque::Failure.backend = Resque::Failure::MultipleWithRetrySuppression

And finally, this is the actual job handling class using resque-status

# app/jobs/notify.rb
class Notify < Resque::JobWithStatus
  extend Resque::Plugins::Retry
  @retry_limit = 3
  @retry_delay = 60

  @queue = :send_notification

  def perform
    ...
    some_heavy_lifting_code
    ...
  end
end

The commands to wake up the workers:

:) QUEUE=* be rake resque:work VVERBOSE=1 -t
:) QUEUE=* be rake resque:scheduler VVERBOSE=1 -t