Free Background Processes on Heroku with One Web Dyno
Sometimes we need to process images, send out emails, or do some other long running process on the server. It is best practice to do these types of jobs in the background so that we don’t tie up the web dyno from serving web requests. Heroku has worker dynos for this purpose.
The problem is that it costs $35/mo per dyno beyond the first free web dyno. That is a bit expensive, especially if you are just building a small site or a toy application. This post assumes that you are already running Puma and Delayed Job.
In puma.rb, within the on_worker_boot block, add
@delayed_job_pid ||= spawn("bundle exec rake jobs:work")
# puma.rb
on_worker_boot do
@delayed_job_pid ||= spawn("bundle exec rake jobs:work")
end
[Update 3/11/2015]
Seems like I get R13 Memory errors when I use this technique with Puma. Switched over to Unicorn and there are no longer memory errors.
# config/unicorn.rb
worker_processes Integer(ENV["WEB_CONCURRENCY"] || 3)
timeout 100
preload_app true
@delayedjob_pid = nil
before_fork do |server, worker|
@delayedjob_pid ||= spawn("bundle exec rake jobs:work")
Signal.trap 'TERM' do
puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
Process.kill 'QUIT', Process.pid
end
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
end
after_fork do |server, worker|
Signal.trap 'TERM' do
puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
end
defined?(ActiveRecord::Base) and
ActiveRecord::Base.establish_connection
end
# Procfile web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
# Gemfile gem 'unicorn'
[Update 3/13/2015]
So I realized why I was getting the memory errors with Puma. Apparently I was running too many threads. I had the workers set to 3 and the max threads set to 16. No idea why. I reduced the workers down to 2 and max threads down to 5, and it seems to have fixed the memory issue.
Text from Heroku docs:
Each worker process used consumes additional memory. This limits how many processes you can run in a single dyno. With a typical Rails memory footprint, you can expect to run 2-4 Puma worker processes on a 1x dyno. Your application may allow for more or less depending on your specific memory footprint. We recommend specifying this number in a config var to allow for faster application tuning. Monitor your application logs for R14 errors (memory quota exceeded) via one of our logging addons or heroku logs.
Here is the new puma.rb config taken from the heroku docs.
workers Integer(ENV['WEB_CONCURRENCY'] || 2)
threads_count = Integer(ENV['MAX_THREADS'] || 5)
threads threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
on_worker_boot do
@delayedjob_pid ||= spawn("bundle exec rake jobs:work")
# Worker specific setup for Rails 4.1+
# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
ActiveRecord::Base.establish_connection
end
Conclusion
Based on what others have said online, the number of stars on github, it seems like puma would be the best option overall. Unicorn is weak against “slow connections”, according to the Heroku docs. I’ve experienced this first hand when trying to upload images from my iPhone app. It will lock up the worker and time out in 30 seconds (or whatever it is set to). Puma doesn’t seem to have this problem.
Sources
https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server
https://coderwall.com/p/fprnhg/free-background-jobs-on-heroku
http://omegadelta.net/2013/06/16/puma-on-heroku-with-mri/