petitviolet blog

    Circuit Breaker in Ruby

    2020-04-16

    RubyFunctional Programming

    I've developed a circuit breaker in Ruby, and published it as rcb gem in RubyGems.org.

    What is CircuitBreaker?

    CircuitBreaker is a well-known pattern to make services stable that was originally introduced by Martin Fowler in 2014.

    It has three states, Close, Open, and HalfOpen.

    • Close is a healthy state and able to run commands
    • Once number of failures reach a given threshold in Close state, circuit breaker goes to Open state
    • In Open state, circuit breaker does not allow to run any command
    • After a while in Open state, goes to HalfOpen state and try a command once
      • if get succeeded, goes to Close state
      • otherwise, back to Open state

    In short, circuit breaker is a pattern to avoid overwhelming services that are unavailable by stopping requests to them. Martin Fowler's post describes what circuit breake is along with Ruby sample codes, but I implemented yet another one for my learning :)

    How to use the rcb gem

    The following shows a simple usage.

    Rcb.configure('api.example.com') do |config|
    config.open_condition.max_failure_count 1
    config.open_condition.window_msec 300
    config.reset_timeout_msec 100
end

response = Rcb.for('api.example.com').run! do
    http_client.get('api.example.com')
end

    Put global configurations using Rcb.configure with tag (e.g. api.example.com, whatever). In Rails app, config/initializers will be a good place to put such configurations.

    How to use rcb gem is that first creating a circuit breaker instance with Rcb.for, and then pass a block to run! method to apply circuit breaker. run! does not restrict commands to HTTP requests or RPC calls. You can pass anything as long as they're Proc objects. As the method has ! suffix, the method invocation may throw an error when given block fails or if the circuit breaker instance's state is Open so that it expects to be rescueed.

    By the way, it's able to create/get a circuit breaker instance with passing configuration to Rcb.for factory method, that is, it can work without set global configuration with Rcb.configure beforehand.

    cb = Rcb.for(:hoge, 
             open_condition: Rcb::OpenCondition.new(5, 1000), 
             reset_timeout_msec: 3000)
cb.run! { 100 }
=> 100

    Configuration

    Rcb has two main configurations.

    • open_condition
      • a condition to decide when trip from close to open state
      • it has max_failure_count and window_msec
        • if failure count reaches max_failure_count within window_msec milliseconds, the circuit breaker trips to open state.
    • reset_timeout_msec
      • if the circuit breaker is in open state and has passed reset_timeout_msec milliseconds, it goes to half-open state

    State trip

    Following shows how state trips.

    >> Rcb.configure(:example) do |config|
     config.open_condition.max_failure_count 2
     config.open_condition.window_msec 1000
     config.reset_timeout_msec 3000
   end
>> instance = Rcb.for(:example)
>> instance.config
=> Rcb::Config(tag: example, open_condition: Rcb::OpenCondition(max_failure_count: 2, window_msec: 1000), reset_timeout_msec: 3000)

>> instance.run! { rand(2) == 1 ? (raise 'Fail!') : :success } rescue :fail
=> :success
>> instance.run! { rand(2) == 1 ? (raise 'Fail!') : :success } rescue :fail
=> :fail
>> instance.state
=> :close

>> instance.run! { rand(2) == 1 ? (raise 'Fail!') : :success } rescue :fail
=> :fail
>> instance.run! { rand(2) == 1 ? (raise 'Fail!') : :success } rescue :fail
=> :fail
>> instance.state
=> :open

>> sleep 3
>> instance.state
=> :half_open
>> instance.run! { rand(2) == 1 ? (raise 'Fail!') : :success } rescue :fail
=> :success
>> instance.state
=> :close

    How to manage state

    Rcb provides a default implementation for managing circuit breakers' states in-memory using class variables. It means that states can be shared only in the same process, cannot be shared with other processes, other machines.

    If you want to share states in a broader context, you can inject your state store implementation to the factory method Rcb.for. It's able to use anything as long as it defines specified interface. For example:

    • use local files to share with other processes
    • use Redis to share with other machines
    • use Thread local variables to share only in the same thread

    About implementation

    The reason why I develop rcb gem is that I want to use Rstructural and guess it enables to coding state management easier.

    As the implementation shows, Rcb::State depends on Rstructural::ADT and Rstructural::Either so much. I've written a blog post about pattern matching in Ruby and the Rstructural gem.

    Empower Pattern Matching in Ruby Quick overview of pattern match in Ruby and Gems to empower it Ruby2.7 provides Pattern Matching feature as an experimental one. https://www.ruby-lang.org/en/news/2019/12/25/ruby-2-7-0-released/ I used to do pattern matching in Scala, so I'm excited to be able to use it in Ruby! tl;dr [Rstructural Gem](https://github.com/petitviolet/rstructural) what I'

    All of Close, Open and HalfOpen are ADT object, and the logic of Close#run uses Either.try, and apply pattern matching on Either::Right and Either::Left to determine the given block get either succeeded or failed. Rstructural::Either.try rescues error thrown from a given block within run!, and it represents either success or failure, so that easy to handle the results using pattern match.

    module Rcb::State
  extend ADT

  Close = data :failure_times do
    def run(config, now: Time.now.utc, &block)
      case Either.try { block.call }
      in Either::Right[result]
        Rcb::Result::Ok.new(self, result)
      in Either::Left[e]
        old_limit = now - (config.open_condition.window_msec / 1000.0)
        refreshed_failure_times = failure_times.filter { |ft| old_limit <= ft } + [now]
        if refreshed_failure_times.size < config.open_condition.max_failure_count
          Rcb::Result::Ng.new(Close.new(refreshed_failure_times), e)
        else
          Rcb::Result::Ng.new(Open.create, e)
        end
      end
    end
  end

  Open = data :since do
    def run(config, now: Time.now.utc, &block)
      # ...
    end
  end

  HalfOpen = const do
    def run(&block)
      case Either.try { block.call }
      in Either::Right[result]
        Rcb::Result::Ok.new(Close.create, result)
      in Either::Left[e]
        Rcb::Result::Ng.new(Open.create, e)
      end
    end
  end

end

    And also defined algebraic data types Rcb::Result consists of Ok and Ng to contain either result or error along with success or failure.

    module Rcb
  module Result
    extend ADT

    Ok = data :new_state, :result
    Ng = data :new_state, :error
  end
end

    It's absolutely easy to understand what Rcb::Result is, because of the Rstructural gem and I'm familiar with functional programming, even if this ruby code does not have annotation of data types of these values.

    Wrapping up

    Implemented a circuit breaker in Ruby using Rstructural gem which offers algebraic data types. It's great experience using pattern matching even if it's untyped language.