2022-01-17
It’s hard for me to believe these words but I pushed Sidekiq’s first commit on Jan 16th, 2012. Ten years ago. The public announcement. One month later. One quarter later.
Some context for those new to this blog: Sidekiq is the most popular background job system for the Ruby programming language.
Every application has tasks which are important: send an email, charge a credit card, reserve inventory for an order, synchronize some data to a 3rd party service, etc.
By making these background jobs, you automatically get error handling and retry, easy scaling across multiple machines, a monitoring UI and lots more.
Nate Berkopec’s incredible Sidekiq in Practice workshop is 10 times cheaper in honor of Sidekiq’s 10th Anniversary. You can buy it for $5 with the discount code SIDEKIQ10.
The discount is good for this week only and ends on 2022-01-24.
Prehistory
Sidekiq did not spring fully formed from the ether.
The first background job system in Ruby which I’m aware of was BackgrounDRb in 2006 from the late, great Ezra Zygmuntowicz.
Ruby on Rails was just taking off in popularity but there was little to no infrastructure to process tasks outside of the HTTP request/response cycle.
The name is a play on DRB, the Distributed Ruby module which ships in Ruby’s standard library, and allows RPC-style calls across machines.
Note the use of the term Worker in the docs; modern Ruby uses the term Job now.
BackgrounDRb’s main issue is that it is not persistent: if the process is shutdown, all executing jobs are lost.
Tobi Lutke, CEO of Shopify, published Delayed::Job in Feb 2008.
DJ persisted job data to the database, allowing for safe, frequent deployments which restarted any DJ processes.
DJ’s main issue is that it targeted MySQL, which was not designed for the access patterns inherent in a queue; it added massive load to the database which limited its scalability pretty severely.
Chris Wanstrath, Founder of GitHub, published Resque in Aug 2009.
Resque moved away from the database and stored job data inside a popular new datastore named Redis.
(Side note: Ezra Zygmuntowicz was the main driver of Redis adoption within the Ruby community and wrote the first Redis client.)
Redis was marketed as a “data structure server” and had native, high performance support for queueing operations, making it far more scalable than DJ.
Resque’s main issue is that
