Due to the recent financial crisis, regulators are more intensely scrutinizing the risk models that financial institutions rely on in their decision-making process. In turn, financial institutions are increasingly worried about how their models and their model management practices will fare in an audit as well as in a dynamic economy. According to a recent survey of FICO banking clients, 98 percent of respondents were undergoing an audit of their model management practices, were anticipating an audit or had already undergone one or more audits. In addition, four out of five respondents said they were concerned about some aspect of regulatory requirements.
Financial institutions have a long, successful history of responding to regulatory requirements, but today there is a tremendous number of regulations covering the full spectrum of banking, from the CARD Act to Dodd-Frank and Basel III. Banks also have to contend with new regulations governing risk models. In April 2011, the Federal Reserve and the Office of the Comptroller of the Currency issued their supervisory guidance on Model Risk Management, calling for the evaluation of models for correctness.
Institutions can meet this heightened compliance challenge by adopting best practices for model management. In addition to lessening regulatory worries, four best practices can improve an institution’s business, allow it to focus on competing in the marketplace and strengthen customer relationships by providing a better understanding of consumer preferences and behavior. Institutions that do not implement best practices may have trouble in meeting regulatory requirements, risking penalties and damage to their reputations. They may also experience increased losses and miss opportunities for revenue growth because their strategies are not equipped to respond to economic changes.
Building a Better Mousetrap
The first best practice is to build models that are robust and transparent, so that it is clear how the model works and how it makes its predictions. The results a model produces should be readily interpretable. The model development process must be disciplined to produce meaningful results and include processes to manage outliers and random noise, as well as to guard against over-fitting and sample bias. Models must be engineered with the business user and business decision in mind.
Second, financial institutions should strengthen the entire model development process. The process should be consistent across projects and across analysts, and should be repeatable. Thorough documentation plays a critical role in making the process transparent. It helps to ensure that the models can be rerun if required, and that the process and its results can be explained to auditors. The process should be collaborative to facilitate learning and to take advantage of the strengths of individual team members.
The third best practice involves implementation without delay. In a recent survey, FICO found that most banks lose anywhere from three to 18 months just getting new models into production due to IT queues, complications in recoding models and testing. By then, the data are old, the competitive landscape has changed and the models that are still waiting to go into production are already obsolete.
It is increasingly important to shorten the time between development and deployment; to put new scoring models into production in a matter of days rather than months.
Finally, financial institutions need to continually review their models and analytics. Sound model management does not end with development or implementation; it requires that models continue to demonstrate stability, accuracy and strong predictive separation between good and bad payers. This can only be demonstrated through validation. Institutions that excel at analytics validate all models in production regardless of type each quarter, or more often in a volatile economy. This includes monitoring score stability, composition and performance through population stability reports, characteristic analyses, final score reports and override reports. The reports provide a common language for communication of results and provide information to guide changes to strategies. The reports also form the foundation of responses to internal and external compliance requirements.
Documenting Every Activity and Analysis
Typically, banks tend to have three major issues with audits: performing the identification and validation at the appropriate sub-population level; preparing the required documentation; and responding in a timely manner to the ad hoc queries that examiners pose, despite limited resources. In an audit, institutions can expect regulators to ask for evidence to support decisioning strategies. The better this is documented, the easier it supports an audit.
It is critical to document every activity and every analysis. Examiners cannot approve what they cannot see or understand. Some activities are internal (e.g., data hygiene), but they all need to be specified and made as consistent as possible. All models must be included in the model management process. Be able to explain the business purpose for the model and the restrictions on its use, as well as the data that are required to support the inputs and outputs.
Today, financial institutions can turn to automated solutions to help ensure compliance. These solutions provide the capability to track, centralize and analyze a wide array of items as well as to simulate any number of related scenarios that regulators might ask about. Beyond audits, such solutions add business value by enabling banks to continually review and re-evaluate their policies and strategies within an ever-changing economic environment.
In the highly competitive financial industry, holding steady carries its own risks. Financial institutions that do not have visibility into the performance of their analytic models may sacrifice revenue, lose good customers, experience increased delinquencies and expose themselves to lawsuits. By not adopting the best model management disciplines, institutions risk difficulties in meeting regulatory requirements, which can lead to disruptions, penalties and damaged reputations. Without total confidence in their models, institutions may be making too many overrides or allowing too much confusion into the decision-making process. By implementing best practices for model management, financial institutions can better manage their regulatory and competitive risks, and ultimately elevate the performance of their decisions.
Sharon O’Connor Clarke is principal consultant in financial services at FICO. Andrew Flint is senior director of product management for FICO’s analytic tools. For more information on FICO’s model management capabilities, please visit www.fico.com.
Copyright (c) June 2012 by BankNews Media