Rate Limiting Work On ActiveRecord::Base With .find_each In Edge Rails

Posted on October 10, 2013 by Carsten Zimmermann

I was working with several maintenance tasks that query external webservices for a collection of ActiveRecord objects. In order to avoid hitting the webservices’ rate limit, we pause every other iteration for a fraction of a second before we continue.

The code looks something like this:

I didn’t like that the information of our rate limit guard clauses was so scattered: there were bits before the block and others in the block. I had the urge to refactor it into a more concise form using a Ruby block itself.

My idea was to create a method that takes the rate limit, the sleep time and an enumeration and then yield the elements of the enum to a block.

I wrapped that method into a module and out came this:

This is how the specs look like:

Unfortunately and unlike other enumerable methods, the batch finder ActiveRecord::Base.find_each does not return an Enumerator when called without a block. Not in Rails 3.2 and not in Rails 4.0.

Luckily, it has been solved in Edge Rails (see 840c552) and I quickly created a Rails 4.1.0.beta app to go for the following code:

ActiveRecord::Base.find_each now returns a proper Enumerator that can be passed into the rate limiter. If you want to learn more about Enumerators in Ruby (or if you’d like to refresh your memory), I can recommend this excellent Ruby Tapas episode by Avdi Grimm.

comments powered by Disqus