What Time Unit Should I Use to View My Approval Rates?

We're working on ways we can help our customers compare the performance of different segments of their payments operation and decided a good starting example would be a customer who uses multiple processors and wants to compare the approval rates between them. 

Some initial questions that came to mind were:

• Can we make statistically significant conclusions about performance between payment service providers (PSPs) if the distribution of the business' transaction volume is not balanced (assuming that the business has correctly configured their connections)?

• How much data do we need to look at? 

• Is "how much data do we need to look at" about the gross total transaction volume over a fixed period of time, or is it a question about quantity of time periods to look over?

This led us to realize that the question of sample sizing for this analysis was conflated with picking the right time frame and time unit for analysis. It’s not useful to think of approval rate as a single number; questions about it are always conditioned on how it behaves over time, which is why it is visualized on a time-series plot. We started out thinking that we were looking for a way to compare multiple approval rates for a performance analysis, but the question of picking the right time-frame and time-aggregation seemed important to work out first. Once we have clarity on this, we will have a coherent basis for comparing metrics.

To illustrate the time-aggregate question, we found a case of a Pagos customer with approval rates that seemed to be much higher when looked at using an hourly view versus day or week. (Note: the time differences that we are exploring in this blog post are true after removing noise around recurring versus non-recurring, retry versus initial transaction, currency, and country.) The hourly view seemed to show an approval rate that is usually around 85-90%, while the daily view showed an approval rate hovering around 70%. Let’s use sample data to explore the ways that hourly and daily approval rates can differ so much and why the right level of aggregation for an analysis of approval rate matters.

The reason the two rates differ so much for this particular customer becomes clear when you look at the total transaction volume per time period. On the hourly view, there is a daily spike in transaction volume that coincides with sharp drops in approval rate. This is an indication that the customer has a daily job to retry transactions.

To help dig deeper in determining the right level of aggregation of approval rate data, we created histograms to illustrate the distribution of the data points from daily and hourly time series Peacock charts (the daily chart shows the distribution of the daily approval rates, and vice versa for the hourly). Each is overlaid with a kernel density estimate, to show the “center of mass” of the distributions. 

Note that there are far fewer histogram bins for the daily chart: even though both charts represent the same period of time, the observations of approval rate are rolled up to different time aggregates, meaning that there are only 16 data points to draw the day-level histogram, and there are 16x24=384 data points to draw the hour-level histogram. These charts show that the time-aggregated approval rates are not normally distributed, which complicates the statistical conclusions we might make when we compare approval rates (we'll talk about this in a future blog post). 

So the correct time unit for analysis of approval rate data depends on your use case. If your intent is to understand the:

• Overall performance of the approval rate because you want to compare it to a benchmark, larger aggregations are better.

• Behavior of the approval rate because you want to be responsive to fluctuations and anomalies before it affects revenue, hour-level is better.

Looking at approval rates over time without taking into account the different sources and business processes that are driving the actual transactions can lead to incorrect conclusions. 

It’s one of the reasons that we make the ability to change time period so easy in Peacock or to allow for flexible windows in Canary lookback triggers—it allows you to select the right lens for the analysis and compare behaviors. 

Returning to the case of our sample customer, the average daily approval rate is very close to the approval rate over the whole data series. This means that day-level (or higher) aggregations on this chart are accurate representations of the overall approval rate of 70%. The larger time aggregates smooth out the volatility of the hour-level measurements. 

Why Does This Matter?

The higher volatility of the hour-level aggregates give an accurate picture of the normal behavior of this customer's payments operation at any given point in time, but because there is a once-daily bulk process that results in a very low approval rate for one hour each day, the hourly approval rate doesn’t accurately reflect the true performance overall - it simply depicts what an average hour looks like. We recommend tagging transactions so that you can clearly see which are first-time, recurring, and retry transactions for even further granularity since transaction types have different success rates. For example, retries will by definition have higher declines; the more you do, the worse your total gets, which will distort your tracking.

The hour-level histogram for this customer shows that most of the time the hourly approval rate is around 90%, with a long left tail that shows how low the hourly approval rate can go for this customer when they process retries. If you were to randomly check the hourly approval rate over the course of a day, most of the time you will observe it to be well above the overall approval rate of 70%. It’s counterintuitive, but both of these approval rate measurements are accurate—they just have different contexts.

Peacock gives you quick and easy ways to experiment with your data and determine what charts and views are most useful for your business. And Canary allows you to set thresholds tailored to your business for alerts on fluctuations or anomalies in your payment processing. Combined, our birds enable you to reduce operating costs, increase revenue, and mitigate unnecessary customer friction.

Interested in learning more about Peacock, Canary, or any of our other payments solutions? Fill out the form below to connect with the Pagos team: