Comparing Threshold Types in Canary: How Do You Know What’s Best?

The power of Canary lies with its triggers. Only once you configure a trigger can you start to receive alerts about aberrational events in your data. As a user, the most helpful thing to understand about trigger configuration is which type of threshold to use: simple or relative. This post will expand upon another one of our blog posts—Setting Thresholds in Canary—to walk you through use cases for simple and relative thresholds. 

Refresher: What’s the Difference Between a Simple and Relative Threshold?

Before we begin, let’s review where you configure triggers in the Canary service panel. To create a new trigger, click Trigger in the navigation bar, then click the Add button in the top-right corner. This opens the Trigger Configuration Panel where you can pick the metric you want to keep an eye on and the Threshold Type. 

To learn more about how to configure a trigger, check out the Pagos Product Documentation.

A simple threshold detects when your data exceeds a consistent minimum or maximum acceptable value. When you create a simple threshold trigger, you have complete control, as you must configure the following fields:

• Threshold Value: Enter the exact numerical value for the chosen metric, beyond which you want to receive notification.

• Direction: Select whether you want Canary to alert you if your metric crosses Above or Below the Threshold Value.

A relative threshold detects when your data deviates from your historical average. Relative threshold triggers require less work on your part; you choose a metric, Canary calculates a threshold range based on where 95% of your historical data sits, and you’ll receive a notification when your data drops below that range. The only field you'll configure is the Look Back window, which specifies the number of days of historical data you want Canary to use when calculating the threshold range.

With definitions out of the way, let’s dig into business cases for when to use each type of threshold.

When Is It Best to Use a Simple Threshold Trigger?

To reframe the above, when you configure a simple threshold, you know the exact threshold value (and whether it’s a minimum or maximum acceptable value) at the time of configuration. You set it yourself. That means that a simple threshold is best used when you know exactly what you’re looking for in your data. Here are three examples of when this might be the case:

Example 1: Your Business Already Has an Established KPI

Simple thresholds are great if your business already has an established KPI that you track on a regular basis. A common KPI for payments organizations is approval rate. If you have already established a KPI for approval rate—say 90%—then you can configure a simple threshold trigger in Canary to alert you when approval rate dips below a threshold value of 90%.

If you didn’t specify filters when configuring the trigger described above, Canary will only alert you when the average approval rate of all of your transaction volume dips below 90%. You may however, have segments of your transaction volume that have different approval rate targets. If, for example, you have multiple processors, you may have a different approval rate KPI for each processor. In this case, you’d want to configure multiple simple threshold triggers: one for each processor.

Example 2: For the Metric In Question, the Industry Has Pre-Set Targets For Standard Performance

If the payments industry already has a standard of performance, you may want to monitor your business relative to that standard with a simple threshold trigger. A perfect example of this is chargeback rate. Networks have established chargeback rate thresholds—generally around 1%—that merchants cannot cross without penalty. If your chargeback rates cross the established threshold, your business could be punished, usually by network or process oversight via a chargeback reduction program and penalty fees paid to the network.

But how do you know that you’re approaching the “danger zone” of a chargeback reduction program? Enter Canary simple threshold triggers. If the network standard is 1% chargeback rates, you could set your trigger to alert you if your chargeback rate increases above a threshold value of 0.8%. That would give you plenty of time to course correct before you meet any repercussions from the industry.

Example 3: You Want to Monitor For a Very Specific Event

Since simple threshold triggers work best when you know what a metric should be, they can also be used to monitor for a very specific event. For example, if your company launches a new product or a promotion, you may expect to see a sharp increase in transaction count. Using the new transaction count value you’ve predicted (and are aiming for), you can configure a simple threshold to alert you when you’ve met that expectation. Then, whenever your data trips the trigger, you’ll have documentation of launch success via the events Canary provides. 

Quick Side Note: If you use Canary this way, make sure you pause any triggers you have already configured for transaction count when there is no product or promotion launch.

When Is It Best to Use a Relative Trigger Threshold? 

When it comes to relative threshold triggers, Canary calculates the threshold range for you. As such, a relative threshold trigger works best when you don’t know exactly what you’re looking for. Let’s take a look at three examples.

Example 1: You Don’t Yet Know What “Normal” Looks Like

In the first simple threshold trigger example above, we explored a situation in which your business had an established approval rate KPI. But what do you do if you’re not sure what your approval rate KPI should be? 

Perhaps your business has a strong US presence, and you’ve decided to launch into the European market. You likely know what your approval rate should be for the US business. A simple threshold—like the example above—would be well-suited for the US approval rate KPI. Your new market, however, is totally different; you may not know what to expect for European approval rates. 

A relative threshold trigger filtered by the "All" parameter is perfect for this situation. You could set up a relative threshold trigger that monitors approval rates for each different currency individually. This trigger will calculate a different threshold range for each currency you accept and alert you when your approval rate for any individual currency drops into the outlying 5% of your historical data. Over time, you could use the relative threshold to inform what is “normal” for your European transactions and establish a KPI.

Example 2: You Think Something Might be Wrong In Your Data, but Aren’t Sure

Again, relative thresholds are great if you don’t know what you’re looking for. Perhaps you’re using both Canary and Pagos’ data aggregation and visualization platform, Peacock. In a Peacock chart displaying data for approval rates by card issuer, you notice that last month, one issuer in particular had a brief, but sharp decline in approval rate. After expanding your investigation, you realize this issuer has had four other brief, but sharp declines. You’re not sure why. 

You decide to monitor this issuer more closely going forward so you can stay informed of the next such decline in real time. However, each incident you uncovered in Peacock declined to different approval rates and for different amounts of time. It’s not immediately clear what is out of bounds for this issuer! Enter relative threshold triggers. You can configure a trigger in Canary with a relative threshold to monitor for extreme drops in that particular issuer’s approval rate. In the future, you can respond more quickly to any problems that issuer may have and use the events as evidence should you want to engage your processor or issuer in a conversation.

Example 3: You Only Want To Be Alerted For Outliers In Your Data

When Canary calculates the threshold range for a relative threshold trigger, it does so using your data’s historical average. Canary then alerts you when your data drops below the range where 95% of your historical data has been. In other words, Canary alerts you to the 5% outliers of your data. 

Let’s say that you want Canary to alert you of something that is truly abnormal, such as an outage in transaction volume or an incident in your technology stack that results in a sharp decline in approval rates. If that’s the scenario you find yourself in, a relative threshold is the right choice.

Are There Going To Be More Threshold Types?

Yes! We are actively working on future iterations of Canary. Our next threshold type will be a predictive threshold where Canary will predict what your metrics should be and alert you if your real data is far from the prediction. Check back for updates on future releases!

Ready to get started?

No-code integration is now open to businesses processing with Braintree, Stripe, or Adyen, and our API is available for use with other processors.