Rate Limiting Work On ActiveRecord::Base With .find_each In Edge Rails
Posted on October 10, 2013 by Carsten Zimmermann
I was working with several maintenance tasks that query external webservices for a collection of ActiveRecord objects. In order to avoid hitting the webservices’ rate limit, we pause every other iteration for a fraction of a second before we continue.
The code looks something like this:
I didn’t like that the information of our rate limit guard clauses was so scattered: there were bits before the block and others in the block. I had the urge to refactor it into a more concise form using a Ruby block itself.
My idea was to create a method that takes the rate limit, the sleep time and an enumeration and then yield the elements of the enum to a block.
I wrapped that method into a module and out came this:
This is how the specs look like:
Unfortunately and unlike other enumerable methods, the batch finder ActiveRecord::Base.find_each
does not return an Enumerator when called without a block. Not in Rails 3.2 and not in
Rails 4.0.
Luckily, it has been solved in Edge Rails (see 840c552) and I quickly created a Rails 4.1.0.beta app to go for the following code:
ActiveRecord::Base.find_each now returns a proper Enumerator that can be passed
into the rate limiter. If you want to learn more about Enumerators in Ruby (or if you’d
like to refresh your memory), I can recommend this excellent Ruby Tapas episode by Avdi Grimm.