Backtesting
Backtesting helps you compare the performance of two or more strategies or workflows over time. By testing strategies against the same population, you can measure which approach drives better outcomes across engagement, payment plans, recovery, and other key metrics.
Use backtesting when you want to validate strategy changes with real account performance rather than relying on assumptions alone.
How Backtesting Works
When you create a backtest, you select the strategies you want to compare. The platform automatically distributes accounts across those strategies and tracks performance over time.
For two-strategy tests, accounts are split 50/50 by default.
Backtesting is designed to help you evaluate which strategy is performing better based on measurable outcomes, not just volume or activity.
Create a Backtest
Prepare your strategies. Create or identify the strategies or workflows you want to compare.
Start a new backtest. Select the strategies to include in the test.
Define your sample size. Specify a target sample size if applicable.
Launch the test. Allow accounts to begin flowing through each strategy.
Once the test is active, the platform continuously tracks results and compares performance across the selected strategies.
What Backtesting Measures
Backtesting compares strategies across measurable outcomes. The specific metrics you focus on will depend on your strategy type and business goals.
Common outcomes include:
Engaged calls - calls where the consumer answered and engaged
Pickup rate - percentage of calls answered
Voicemail rate - percentage of calls resulting in voicemail
Payment plans - number or rate of payment plan enrollments
Promises - promise-to-pay arrangements
Recovery performance - amount recovered or recovery rate
Reading Results
As data is collected, the platform summarizes results for each strategy and identifies which strategy is currently leading.
Winner Designation
When one strategy outperforms the others with sufficient statistical support, the platform identifies it as the current winner. This helps teams quickly understand which strategy is producing stronger results.
Statistical Analysis
Backtesting includes statistical analysis to determine whether observed differences are meaningful.
Available metrics:
P-value - measures whether the difference between strategies is likely due to chance
Confidence - shows the statistical confidence level for the current result
Chi-square - provides the test statistic used in the analysis
Significance - indicates whether the current result is statistically significant
If significance has not yet been reached, the platform may recommend continuing to collect data.
Segmentation
Backtesting results may include segmented views, such as by state or other dimensions. This helps you understand whether one strategy performs better for a specific subset of accounts.
Segmented analysis is useful when overall performance is mixed but a strategy is clearly stronger within a particular segment.
Ongoing Tests
Backtests are ongoing by design. As more accounts move through each strategy, the platform updates results automatically.
This makes it possible to:
monitor tests over time
validate whether early results hold as sample sizes grow
identify when a leading strategy becomes statistically significant
Best Practices
Use backtesting when you want to:
compare two collections strategies
test different servicing approaches
measure the impact of workflow changes
evaluate strategy performance across different portfolios or segments
For reliable results:
Make sure the strategies being compared are intended for similar account populations
Allow the test to run long enough to collect a meaningful sample
Wait for statistical significance before declaring a winner
Review segmented results to identify population-specific differences
Important Notes
Account split. For two-strategy tests, accounts are split evenly (50/50) by default.
Results change over time. Early leaders may shift as more data is collected. Treat early results as directional until statistical confidence is reached.
Segmented performance may differ. A strategy that performs well overall may underperform in specific states or segments. Review segmented views before scaling.
Summary
Backtesting gives you a structured way to compare strategies using real outcomes. By measuring performance over time and applying statistical analysis, you can make more informed decisions about which strategies to expand, refine, or retire.