Reliably Deploying Rails Applications: Hassle free provisioning, reliable deployment (2014)

18.0 - Sidekiq

Overview

Sidekiq is a simple and extremely memory efficient background job processing library for Ruby. It’s not Rails specific but has very tight Rails integration available. It is a drop in replacement for Resque and offers huge memory savings. In this chapter we’ll look at how to automatically start and restart Sidekiq workers when we deploy and use Monit to ensure it continues running.

Sidekiq Version 3

This chapter has been written for Sidekiq 2.x. Sidekiq 3 has been recently released with some significant improvements. This chapter will be updated for Sidekiq 3, until then this process should work as long as the capistrano-sidekiq gem is included.

There’s more about the changes in version 3 here http://www.mikeperham.com/2014/03/28/sidekiq-3-0/.

Capistrano Integration

Sidekiq 2 provides in built Capistrano integration. In Sidekiq 3 this has been factored out into a gem capistrano-sidekiq. To include this functionality simply add the following line to your Capfile:

1 require 'capistrano/sidekiq'

This will automatically add the following hooks:

1 after 'deploy:starting', 'sidekiq:quiet'

2 after 'deploy:updated', 'sidekiq:stop'

3 after 'deploy:published', 'sidekiq:start'

This means that when the deploy starts (deploy:starting), the following will be issued to instruct the Sidekiq worker process not to start processing any new jobs:

1 bundle exec sidekiqctl quiet SIDEKIQ_PID

Once the code has been updated, the following will be issued to stop the worker process:

1 bundle exec sidekiqctl quiet SIDEKIQ_PID

And finally once the new version of the app has been published, a worker process using the new codebase will be started with:

1 bundle exec sidekiq ...(options)

This is all completely transparent to us and requires no additions to the deploy code other than requireing the Capistrano tasks in our Capfile.

The reason for doing this in three steps rather than one is to allow Sidekiq as much time as possible to gracefully finish processing any existing jobs.

We want to ensure that Sidekiq stores a Pidfile with a known name so that we can use Monit to keep track of its status. Furthermore we want to ensure that it follows our directory structure convention of storing all pid files in APP_ROOT/tmp/pids. Therefore we add the following to ourdeploy.rb:

1 set :sidekiq_pid, "#{current_path}/tmp/pids/sidekiq.pid"

Init Script

Although the Sidekiq process will be automatically started when we deploy, we want an Init Script to allow us to manage it with Monit.

To enable the init.d script provided in the sample configuration, first add sidekiq_init.sh to the :config_files array in deploy.rb.

Then add sidekiq_init.sh to the :executable_config_files array in deploy.rb.

Finally add

1 {

2 source: "sidekiq_init.sh",

3 link: "/etc/init.d/sidekiq_{{full_app_name}}"

4 }

to the :symlinks array in deploy.rb.

You’ll then need to run cap STAGE deploy:setup_config to update the remote server with the init script.

Monit Config

Add the followed to then end of your config/deploy/shared/monit.erb

1 check process sidekiq_worker

2 with pidfile <%= current_path %>/tmp/pids/sidekiq.pid

3 start program = "/etc/init.d/sidekiq_<%=application%>_<%= fetch(:rails_env)%\

4 > start"

5 stop program = "/etc/init.d/sidekiq_<%=application%>_<%= fetch(:rails_env)%>\

6 stop"

You’ll then need to run cap STAGE deploy:setup_config to update the remote server with the new monit configuration.

All materials on the site are licensed Creative Commons Attribution-Sharealike 3.0 Unported CC BY-SA 3.0 & GNU Free Documentation License (GFDL)

If you are the copyright holder of any material contained on our site and intend to remove it, please contact our site administrator for approval.