This movies from using multiple redis calls in ruby to redis server
side lua scripts for the rolling counter and the math contained in
FailurePercentCounter. This provides atomic operation over all the set
of redis calls and should provides efficiency over multiple separate
call for the data.
closes CNVS-34236
ref CNVS-34031
test plan:
- Ensure canvas is configured to use redis
- In the rails console do the following:
Setting.set('service_qatesting_timeout', 1)
Setting.set('service_qatesting_timeout_protection_method',
'percentage')
Setting.set('service_qatesting_min_samples', 10)
- The following must be completed in the rails console in under a
minute:
Canvas.timeout_protection('qatesting') { }
Canvas.timeout_protection('qatesting') { }
Canvas.timeout_protection('qatesting') { }
Canvas.timeout_protection('qatesting') { sleep 2 }
Canvas.timeout_protection('qatesting') { }
Canvas.timeout_protection('qatesting') { sleep 2 }
Canvas.timeout_protection('qatesting') { }
Canvas.timeout_protection('qatesting') { }
Canvas.timeout_protection('qatesting') { }
Canvas.timeout_protection('qatesting') { }
Canvas.timeout_protection('qatesting') { }
Canvas.redis.ttl("service:timeouts:qatesting:percent_counter:protection_activated")
- Note that after each 'sleep' above that an error report of type
'service_timeout' was generated
- Note that after 10 samples, we went into timeout protection by
the log message of "Skipping service call due to error count: qatesting 0.2"
- Note that the Canvas.redis call returns a number between 0 and 60
Change-Id: Ic04eaab4edb49518e47538feda06dd32a32b49ec
Reviewed-on: https://gerrit.instructure.com/99764
Tested-by: Jenkins
Reviewed-by: Shahbaz Javeed <sjaveed@instructure.com>
Reviewed-by: Spencer Olson <solson@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
QA-Review: KC Naegle <knaegle@instructure.com>
Product-Review: Keith T. Garner <kgarner@instructure.com>
080d4e2e26