Audit Alert: The rule has exceeded its error rate quota and is being throttled

You or your Organization Administrator (Org Admin) may see an error like the following in either the error log for the namespace where it happened or the “root” namespace used by the Org Admin:

Message: 
Audit Alert: The rule <your_rule> has exceeded its error rate quota and is being throttled.

That error is produced by the VANTIQ Error Circuit Breaker.  The goal of the circuit breaker is to stop processing failing rules/procedures since they cannot be correct.  If you have a coding bug it might cause a rule or procedure to ALWAYS generate an exception.

The Error Circuit Breaker shuts them down!

When you see this audit alert you should fix the problem ASAP as this cannot be correct behavior.

The configuration settings behind the VANTIQ Error Circuit Breaker along with their default settings are:

  • sample (20) – requests that must be seen before inspection is initiated
  • failurePercent (80%) – if more than 80% of inspected requests generate exceptions​ then throttle the rule/procedure
  • retrySample (2) – how many invocations must be seen before re-evaluating result
  • retryAfter (1 minute) – how long to wait before trying again after the breaker trips
  • auditFrequency (10 minutes) – how often audit messages can be produced for a given audit condition
  • errorReportingFrequency (30 minutes) – how often the same error report for the same violation type is reported

IMPORTANT: If you see error messages in the error log, do not ignore them as each message may represent many of these failures.​

You should never change the above settings without consulting with VANTIQ Technical Support.

The audit alerts get bubbled up to the Org Admin level so that the Org Admin can notice the problem even if they do not go into the problem namespace (i.e. a copy of the audits is stored in the Org Admin namespace as well as the problem namespace).

As stated above, you should fix things BEFORE they cause a real problem.

 

Posted: August 12, 2020 at 11:41 pm
Replies: 0
Viewing 1 post (of 1 total)
  • You must be logged in to reply to this topic.
© Vantiq 2024 All rights reserved  •  Vantiq Corporation’s Privacy Policy and Cookie Policy