Load-balancers.

Almost every platform uses them these days to distribute the incoming requests between applications, to make sure one node or cluster does not get overloaded and that all users experience an acceptable performance.

The downside of these load-balancers however is that they need to send out some kind of keep-alive messages to the back-end, in order to verify whether the various endpoints they are balancing are still reachable. In our case, this was done by triggering a specific URL and check if we get an HTTP 200 response.

Now without disclosing too much of the internals, we track every web-request, rely on sessions and don’t want to pollute our database with information that’s not relevant to the system. So we originally had the idea to filter out all load-balancer requests from the controllers. Sounds perfectly okay so far, right? Except code wise this was a nightmare. We had a bunch of filters, an API to support and sessions to clean up. This resulted in the ApplicationController¬†becoming a mess of hooks, cleanup code and basically undoing everything Rails did when a new request came in.

When another change-request came in to filter out session data from Redis for the load-balancer, I started asking myself the question: Do we even need to hit the Rails stack in order to know our application is up and running? The answer to that, is no. You don’t need to hit Rails to know whether your application is up and running.

Rails has this feature called Rack Middleware. Creating a custom middleware that responds simply to the load-balancer requests, we created a simple solution that returns an empty HTTP 200 response for the load-balancer, but processes all other requests further up the stack.

  • The Rails stack is never called. This means no cleaning of sessions, data or implementing dozen’s of hooks and if-statements to filter out load-balancer requests.
  • The middleware runs before everything else, so we don’t even need to load all code or triggers
  • The load is removed from the database and Redis system as the calls are never made
  • A lot of the code is reduced to a single module.
module Middleware
  class LoadBalancer
    def initialize(app)
      @app = app
    end

    def call(env)
      params = { url: env['REQUEST_URI'], ip: env['REMOTE_ADDR'] }
      return blank_response if LoadBalancer.skip_web_request?(params)
      @app.call(env)
    end

    def blank_response
      [200, { 'Content-Type' => 'text/html', 'Content-Type' => 0 }, ['']]
    end

    class << self
      def skip_web_request?(url:, ip:)
        skip_web_ips.include?(ip) && skip_web_urls.include?(url)
      end

      def skip_web_ips
        Settings.proxy.trusted_ips
      end

      def skip_web_urls
        ['http://our_fancy_balance_url']
      end
    end
  end
end

So basically what this class does it return an empty HTTP 200 response if the request matches our Load-Balancer request or delegate it up the stack in all other cases.

To get the load-balancer to work before everything else in your stack, you need to create an initializer for Rails and add the following code:

Rails.application.config.middleware.insert(0, ::Middleware::LoadBalancer)

 

Advertisements