Nepal on Rails

Setting up Resque, Status, Retry, Multiple failure backend with Rails 3

Setting up Resque is easy-peasy with Rails app. But after adding couple of plugins, things gets messy and sometimes you just can’t figure out that its really working as it supposed to.

Later I added resque-status to monitor the health of the jobs. Then added resque_mailer to send emails asynchronously. Then I added resque-retry to retry the jobs themselves to retry 3 times in 60 secs of interval. After deploying to production, it was being hard to assure/verify that its actually working as it was supposed to. Then, had to setup the resque jobs to send errors to hoptoad/airbrake that occurs inside the jobs themselves.

So, after doing the back-n-forth manual testing, the following is the setup/configurations that did exactly the way the application wanted.

Following are the gems used in the app

# Gemfile
gem 'resque', '~> 1.19.0', :require => 'resque/server'
gem 'resque_mailer'
gem 'resque-status'
gem 'rufus-scheduler'
gem 'resque-retry'

Configuration for the hoptoad/airbrake

# config/initializers/hoptoad.rb
HoptoadNotifier.configure do |config|
  config.api_key = "71a95c9cb6552275638f0600b7e6b65c"
  config.development_environments = ['test'] # Environments added here won't be sent
  config.host    = 'millibrake.cloudfoundry.com'
  config.port    = 80
  config.secure  = config.port == 443
end

You might stumble upon what that millibrake.cloudfoundry.com is? Well its just a clone of Hoptoad at CloudFoundry. You can setup your own following this post or you can just use the hoptoad one.

This is initializer file for resque and its addons

# config/initializers/resque.rb
require 'resque/server'
require 'resque/status_server'
require 'resque/job_with_status'
require 'resque-retry'
require 'resque-retry/server'

require 'resque/failure/multiple'
require 'resque/failure/hoptoad'
require 'resque/failure/redis'

config = YAML::load(File.open("#{Rails.root}/config/redis.yml"))[Rails.env]
Resque.redis = Redis.new(:host => config['host'], :port => config['port'], :thread_safe => true)

# Resque Authentication protection (view at http://)
Resque::Server.use Rack::Auth::Basic do |username, password|
  username == config['username']
  password == config['password']
end

Resque::Status.expire_in = (24 * 60 * 60) # 24hrs in seconds

# Exclude sending actual emails in these environments
Resque::Mailer.excluded_environments = [:test, :cucumber, :development]

# Failure Backends Setup
Resque::Failure::Hoptoad.configure do |config|
  config.api_key = HoptoadNotifier.configuration.api_key
end

Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis, Resque::Failure::Hoptoad]
Resque::Failure.backend = Resque::Failure::MultipleWithRetrySuppression

And finally, this is the actual job handling class using resque-status

# app/jobs/notify.rb
class Notify < Resque::JobWithStatus
  extend Resque::Plugins::Retry
  @retry_limit = 3
  @retry_delay = 60

  @queue = :send_notification

  def perform
    ...
    some_heavy_lifting_code
    ...
  end
end

The commands to wake up the workers:

:) QUEUE=* be rake resque:work VVERBOSE=1 -t
:) QUEUE=* be rake resque:scheduler VVERBOSE=1 -t