Sidekiq Pro: Expiring Jobs and Filtering in the Web UI

You might have a job that has a limited usefulness. With the open source version of Sidekiq, once this job is enqueued, it's going to run whether it provides a benefit or not. With Sidekiq Pro, you can make use of the "Expiring Jobs" feature to account for this. Let's check it out.

Project

We'll use the application from the last episode, just to have a starting point. Let's assume we're building an auction application, and we want to notify someone that their bid has just been beaten.

class NotificationWorker
  include Sidekiq::Worker
  sidekiq_options expires_in: 1.hour

  def perform(email)
    UserMailer.auction_notice(email).deliver_now
  end
end

This will set the default expiration time for this job to 1 hour from its creation time. This means that if the job doesn't start being worked within an hour of being enqueued, it won't be worked. It's important to know that if a job isn't worked due to expiration, it's still considered successful for the purpose of batches. The semantics here are that it is acceptable for the job to not be worked, so failing to be worked is not a failure condition.

Here we've defined a static expiration time. You can also specify an expiration time when you enqueue the job.

In order for expiration to work, you need to enable the expiry feature in an initializer:

vim config/initializers/sidekiq.rb

require 'sidekiq/pro/expiry'

Now, let's open up a console and enqueue this job:

rails c

We'll push a new job that expires in 1 second. Here, I'm showing that you can override the default provided in the class's sidekiq_options call:

NotificationWorker.set(expires_in: 1.second).perform_async('josh@smoothterminal.com')

Here we pushed the job in with essentially an immediate expiration. What happens now when we run our sidekiq worker?

2016-05-26T17:24:13.349Z 28387 TID-cwar4 NotificationWorker JID-35b70d2b408abe8651d8c07f INFO: start
2016-05-26T17:24:13.349Z 28387 TID-cwar4 NotificationWorker JID-35b70d2b408abe8651d8c07f INFO: Expired job 35b70d2b408abe8651d8c07f
2016-05-26T17:24:13.350Z 28387 TID-cwar4 NotificationWorker JID-35b70d2b408abe8651d8c07f INFO: done: 0.001 sec

OK, so this job was already expired when the worker got to it, so it popped it off the queue without running it. This is enabled by the expiry middleware that we added via our sidekiq initializer. Sidekiq supports the notion of middleware that can modify how it works, and the expiry middleware tells it to discard expired jobs without executing them. I love systems with middleware, so learning that this was how Sidekiq provides itself an extension point made me very happy.

Filtering with the Web UI

So that's how Sidekiq Pro's expiry middleware works. That was super quick, so we'll also have a look at another topic in this video - filtering in the Web UI. In order to really see what it's used for, we're going to do some weird things.

First, let's make sure Sidekiq is stopped.

Next, we'll go ahead and enqueue a lot of these jobs:

rails c

100.times{ NotificationWorker.perform_async('josh@smoothterminal.com') }

Now we'll drop out and add a second parameter to the worker's perform method:

vim app/workers/notification_worker.rb

class NotificationWorker
  include Sidekiq::Worker
  sidekiq_options expires_in: 1.hour

  def perform(email, amount)
    # And we'll update the method body to something that can actually work...we
    # had no UserMailer.
    puts "Notifying #{email} about being outbid by $#{amount.to_f}"
  end
end

So we're adding a parameter so that the user can find out how much they were outbid by. We'll add some more jobs using this updated method.

rails c

100.times{ NotificationWorker.perform_async('josh@smoothterminal.com', 20) }

Finally, we decided that they need to know who outbid them, so we update that as well:

class NotificationWorker
  include Sidekiq::Worker
  sidekiq_options expires_in: 1.hour

  def perform(email, opponent, amount)
    # And we'll update the method body to something that can actually work...we
    # had no UserMailer.
    puts "Notifying #{email} about being outbid by $#{amount.to_f} by user #{opponent}"
  end
end

And we add some jobs:

100.times{ NotificationWorker.perform_async('josh@smoothterminal.com', 'evil@smoothterminal.com', 20) }

Now we'll start sidekiq and rails, and visit the admin.

We have a lot of jobs failing for different reasons. Let's find the ones that we gave 1 argument to and just delete them.

((( Filter by "given 1, expected 3" )))

Now we can see the remaining ones were given 2 arguments and expected 3. This one, we'll fix with an update to our code:

class NotificationWorker
  include Sidekiq::Worker
  sidekiq_options expires_in: 1.hour

  def perform(email, opponent, amount=0)
    if opponent.is_a? Integer
      # Oops this old code, this is really the amount.
      amount = opponent
      opponent = "unknown@smoothterminal.com"
    end
    puts "Notifying #{email} about being outbid by $#{amount.to_f} by user #{opponent}"
  end
end

We'll restart sidekiq and rails, and retry these jobs. And they succeded.

Summary

In today's episode, we saw how to use Sidekiq Pro's Expiring Jobs feature to produce jobs that become irrelevant after a certain amount of time and consequently need not be run if they can't be run in the time allotted. This can help you avoid sending stale or useless messages to your users.

We also looked at how to use the Sidekiq Pro's Web UI filtering to manage retries even in the face of quite a few of them to sort through. I hope you enjoyed it; see you soon!

Resources

• Expiring Jobs on the Wiki
• knewter/sidekiq_batches - Tagged with before_episode_004.4