The financial paradox: Can an expected credit loss model be well calibrated and still be poor at supporting decisions?
In recent projects with Brazilian financial institutions, we found models that produced reasonable aggregate estimates but struggled to distinguish higher-risk from lower-risk loans. In one case, a rule identified 80% of future defaulters but also flagged 73% of customers who would remain current as problematic. In another, the correlation between predicted and realized LGD was virtually zero, indicating that the model could barely rank loans by risk.
This matters when reviewing models implemented under resolutions such as the Brazilian CMN Resolution 4,966/2021, which is aligned with IFRS 9. Before discussing algorithms, we need to establish whether the underlying economic outcome was properly defined, whether the data allow us to reconstruct it, and, above all, whether the model can differentiate risk at the individual loan level.
That is what can turn a model developed to meet a regulatory requirement into an effective decision-making tool.
Before improving the model, we had to rebuild the problem.
LGD measures the share of credit exposure lost when a loan defaults. In one such project, we expected to find opportunities to improve the existing model. Instead, we discovered that the most significant problem lay in how LGD itself was measured.
The previous model tracked recoveries for just over a year and applied treatment rules that affected the very LGD it was supposed to estimate. Our first decision, therefore, was to stop debating algorithms and rebuild the target variable.
We examined when recoveries actually occurred, the marginal benefit of extending the workout period, what were the recovery costs, and how the time value of money should enter the calculation. We extended the workout period to 72 months.
That change solved one problem but created another: more recent defaults did not yet have six years of history. Rather than simply exclude them, we explicitly addressed recoveries that had not yet been observed before returning to modeling.
In one retail segment, MAE fell from 37% to 20%, a 48% reduction. More importantly for risk differentiation, the Spearman correlation between predicted and realized LGD increased from virtually zero to 0.94.
The lesson from this project is straightforward: a better algorithm cannot fix a poorly constructed target.
Complexity is worthwhile only when it delivers results.
Explainability comes up in nearly every conversation we have with risk teams. That is why we do not automatically select the most complex algorithm.
We test different approaches and ask a specific question: what do we gain from the added complexity?
It might be a meaningful reduction in out-of-sample error, better differentiation across loans, or another measurable improvement. If the gain is small, explainability, stability, and ease of maintenance carry more weight.
This also changes how we build models.
We work alongside each institution’s teams from the earliest structural decisions: definitions of default and cure, segmentation, variable selection, workout periods, and validation strategy. This brings in business knowledge that may not be captured in the data while enabling the responsible team to understand why the model produces a given estimate.
We do not want to deliver a black box that only the consulting team can explain.
What we found and what changed:
- LGD with limited ability to differentiate among loans
We revisited the target, workout period, segmentation, and models. MAE fell from 37% to 20%, while Spearman correlation rose from virtually zero to 0.94. - Credit decision rule rejecting too many creditworthy customers
We developed loan-level PD estimates and revised the decision cutoff. While continuing to identify 80% of future defaulters, the false-positive rate fell from 73% to 31%. - Model complexity making explanation and maintenance difficult
We required measurable gains to justify added complexity and developed the models with the internal team. The result was a model that the people responsible for using and defending it could understand.
A better risk model can mean extending more credit
That was the counterintuitive finding from another project.
The institution’s existing rule identified 80% of future defaulters. The problem was that, to do so, it also flagged 73% of customers who would remain current as problematic.
Our PD model achieved an AUC of 0.82. Holding recall at 80%, the false-positive rate could fall to 31%, a reduction of 42 percentage points.
At a different cutoff, keeping the same volume of rejections, the model could increase the share of future defaulters identified to 98%.
When we presented the results to the institution’s leadership, the conversation moved beyond AUC and calibration.
The client itself identified other uses for PD: making credit committee decisions more objective, helping justify approvals and denials, reducing the influence of personal discretion, and providing customers with a more informative explanation when credit was denied.
This is where a compliance project starts creating business value.
Banks have already invested substantially in meeting CMN Resolution 4,966. The same data, models, and infrastructure developed for regulatory compliance can support credit approvals, credit limits, pricing, collections, and portfolio management.
But there is a condition: the model must differentiate risk, not merely reproduce the average loss.
For an institution that already has its credit risk models in production, we recommend starting the review with a simple question:
When we rank loans from lowest to highest predicted PD or LGD, do observed losses increase as they should?
If the model gets the average right but cannot rank risk, the review is not finished.
Glossary:
- CMN Resolution 4,966/2021: a regulation issued by Brazil’s National Monetary Council that aligns accounting for financial instruments in Brazil with IFRS 9 and requires provisions based on expected credit loss.
- Expected credit loss (ECL): an estimate of the amount a financial institution expects to lose on a loan, calculated using PD, LGD, and EAD.
- PD (Probability of Default): the probability that a borrower or loan will default over a defined time horizon.
- LGD (Loss Given Default): the share of a credit exposure a financial institution ultimately loses after a default, net of recoveries.
- EAD (Exposure at Default): the amount exposed to credit risk at the time of default.
- Default: the point at which a loan is classified as being in default under the institution’s and applicable regulatory criteria.
- Cure: the return of a defaulted loan to performing status.
- Workout: the period during which a financial institution tracks recoveries after default to measure realized losses.
- Calibration: a model’s ability to estimate the portfolio’s average level of risk or loss accurately.
- Discrimination: a model’s ability to distinguish higher-risk from lower-risk loans.
- AUC (Area Under the Curve): a discrimination metric ranging from 0.5 (random classification) to 1 (perfect separation) between performing and defaulting borrowers.
- Spearman correlation: measures whether the ranking of model predictions aligns with the ranking of observed outcomes; values closer to 1 indicate better risk ranking.
- MAE (Mean Absolute Error): the average absolute difference between predicted and realized values.
- Recall (Sensitivity): the percentage of future defaulters that the model correctly identifies.
- False positive: a customer classified as high risk who, in fact, remains current.

BLOG





