Okay,

Based on the title, you’d think this was supposed to be something simple, right? Well, I can tell you that it completely depends on the gems you are using to get all your entries properly configured with a reasonable TTL, and not fall back on the defaults of one year and infinite.

::Redis::Semaphore

One of the first gems we use in our application is the ::Redis::Semaphore gem. We use this gem to create, as the name implies, semaphore objects in Redis to prevent the simultanious execution of background jobs in our Sidekiq framework.

The documentation of this gem makes mention of the :expiration key that can be set. They also warn that this might be dangerous because a key could expire during execution of the process. While this might be true, this luckily was a non-case in our situation.

Basically, when a key receives a TTL, the timer starts running as soon as the key is created in Redis. But the Redis documentation clearly states that the usage of GET, SET and GETSET refreshes the TTL on the key, and tests have shown this. So with the way we use the semaphore keys, our key should not expire during execution of the process. The snipper below shows how to use the setting when creating a semaphore:

::Redis::Semaphore.new("our key", redis: Connection.redis_client, expiration: 7.days.to_i)

Rails Cache

The second place where we use the Redis for caching purposes is the overal Rails cache. In order to make this happen, we relied on the gem redis-rails. This game makes it possible to use Redis as caching back-end for the Rails framework. The configuration is done in your application.rb and is pretty straightforward:

# Redis Cache Configuration.
# With the new TTL settings, keys are now automatically expired after 7 days.
config.cache_store = :redis_store, Chamber[:redis][:cache], { expires_in: 7.days.to_i }
config.session_store :redis_store, redis_server: Chamber[:redis][:cache], key: Chamber[:redis][:session_key], expires_in: 1.year

Exceptional cases

Of course, it wouldn’t be a real application if there weren’t any exceptions to using the standard configuration. There’s an edge case in our application where we do not wish to expire the data stored inside Redis, but how do we do that when the global cache has now been set to 7 days?

Easy, you simply remove the TTL in the caching call. When the TTL is explicitly set to nil, it will be translated to -1 by Redis, which means indefinitely:

Rails.cache.fetch(cache_key.to_param, expires_in: nil) { yield }

Geocoder

This is where is gets more interesting. The geocoder gem allows you to make calls to various endpoints to have an address transformed into geo-coordinates. They way we used this gem, was that we store the returned information in Redis and have it as a faster lookup when the address matches, reducing the load on the endpoints we used.

The gem itself does not supported setting a TTL on the keys it uses. Having it configured to store everything in Redis, we wrote a small wrapper that mimics the behaviour of the Geocoder and sets the desired TTL for us:

module Geocoder
  class AutoExpireRedisCache
    # Initializes the cache using the actual store and TTL as arguments.
    # By default, keys are expired after a week.
    def initialize(store, ttl = 7.days.to_i)
      @store = store
      @ttl = ttl
    end

    # Looks up the value using the provided URL as key.
    def [](url)
      @store.[](url)
    end

    # Store the provided value, using the URL as key.
    # The stored key is expired after the defined TTL.
    def []=(url, value)
      @store.[]=(url, value)
      @store.expire(url, @ttl)
    end

    # Returns all keys currently used by the store.
    def keys
      @store.keys
    end

    # Deletes the specified key from the store.
    def del(url)
      @store.del(url)
    end
  end
end

::Redis::Rack::Cache

This was the hardest gem to have it follow our TTL policy for Redis keys. The gem is basically a hook for Rack to store it’s cache data inside Redis. Unfortunately we do not use this as it should be used, in that way that we never call the cache_for method in our controllers. So throughout the entire application we rely on the default settings of this gem.

This gem has a constant called ::Rack::Cache::Redis::DEFAULT_TTL which is set to 1 year. A whole year is pretty long to keep cached pages and metadata for the application. Looking through the documentation of both Redis, Rack and this game, we came across an initial solution:

config.action_dispatch.rack_cache = {
  metastore: "#{Chamber[:redis][:cache]}/metastore",
  entitystore: "#{Chamber[:redis][:cache]}/entitystore",
  default_ttl: 7.days.to_i,
  use_native_ttl: true
}

Unfortunately this doesn’t work at all. This ends up with the weird behaviour that everything gets stored for over a year because of the constant defined by the framework. The gem offers no configuration options to overwrite this behaviour.

So what does every good Ruby programmer do?

# HACKY TIME!
# We redefine the constant of the Redis Rack Cache to be 7 days instead of one year.
# Since this gem doesn't like to be configured, Oli forces it to be configured using ruby sugar.
::Redis::Rack::Cache.send(:remove_const, :DEFAULT_TTL)
::Redis::Rack::Cache.send(:const_set, :DEFAULT_TTL, 7.days.to_i)

We injected this snippet before the configuration part in our application.rb, overwriting the constant with a setting that’s more to our liking. While it’s not the most elegant solution it does get the job done:

➜  ~ redis-cli
127.0.0.1:6379> select 1
OK
127.0.0.1:6379[1]> INFO KeySpace
# Keyspace
db0:keys=2,expires=0,avg_ttl=0
db1:keys=51,expires=51,avg_ttl=28443512775
127.0.0.1:6379[1]> TTL metastore:4453e4e41864af53bcce5e2c3dffe719a92374ae
(integer) 68400 # 7 days.

Of course this requires you to initially flush all keys when these settings are applied, but we achieved our desired goal : Keep the memory footprint of Redis in line.

Advertisements