Model Risk Management for Financial Institutions: Where AI Security Fits

Financial institutions in Australia have maintained model risk management frameworks for many years. The framework disciplines around model validation, performance monitoring, and governance were developed in response to the risks that statistical and quantitative models introduced into credit decisioning, market risk management, and capital calculation. Those frameworks are mature, and the organisations that maintain them are experienced in applying them to new model types as they emerge.

AI systems deployed in decision-making roles within financial institutions fall within existing model risk management obligations. That much is now clear from APRA's signals on AI risk management. What is less clearly understood in many institutions is where AI security fits within the model risk management framework: how AI-specific security risks relate to the model risk dimensions that existing frameworks address, and what new assessment activities are required that model validation as traditionally practised does not cover.

AI security risks that model risk frameworks already partially address

Traditional model risk management addresses a range of risks that are directly relevant to AI security. Model performance monitoring, for example, is the mechanism by which an institution detects when a deployed model is producing outputs that diverge from its expected behaviour. This is the same mechanism that would detect certain adversarial attacks on an AI system: if the model's outputs change in unexpected ways, the performance monitoring framework should flag it.

Model validation, as practised in most institutions, includes assessment of model limitations, sensitivity analysis, and review of the data used to develop the model. These activities overlap with AI security assessments of training data quality, data poisoning risk, and model robustness to distributional shift. Institutions that have mature model validation capabilities are not starting from scratch when it comes to AI security assessment; they are extending their existing practice to address attack vectors and failure modes that traditional model validation does not specifically target.

AI security risks that model risk frameworks do not yet adequately address

The adversarial dimension of AI security is the area where model risk frameworks most frequently have gaps. Traditional model validation asks whether a model performs well on representative data. Adversarial security testing asks whether an attacker who deliberately crafts inputs to manipulate the model's output can succeed. These are different questions, and the second is not a natural extension of the first for most model validation teams.

Prompt injection, evasion attacks, and model extraction are specific to AI systems and fall outside the scope of conventional model validation. An adversarial attack on an AI-powered credit decisioning system might not change the model's average performance on standard test sets but could consistently produce incorrect outputs for a specific class of borrower who knows how to exploit the model's decision boundary. Detecting and preventing this requires adversarial testing capability that most model risk functions do not currently have in-house.

Integrating AI security into the model risk framework

The most efficient approach for most institutions is to extend the existing model risk framework to incorporate AI security requirements, rather than building a separate AI security programme in parallel. This means adding AI-specific requirements to model tiering criteria, so that models with material adversarial attack surfaces receive additional validation scrutiny. It means adding adversarial testing to the pre-deployment validation requirements for AI models. And it means ensuring that model performance monitoring is designed to detect not only statistical drift but also attack signatures.

The accountability structures within model risk management provide a natural home for AI security governance. The model owner accountable for a credit model is also accountable for that model's security. The model validation function responsible for validating the model's performance is also responsible for assessing its security properties. Integrating AI security into model risk does not require creating new accountability structures; it requires ensuring that existing accountability holders have the information and capability to discharge their extended responsibilities.

What APRA expects and how to demonstrate it

APRA's expectations for AI risk management in regulated entities are expressed in terms of outcomes rather than specific processes. Entities should be able to demonstrate that material AI systems have been risk assessed, that accountability is clear, and that ongoing monitoring is in place. Within a model risk management framework, this translates to:

• AI systems in scope for the model risk framework are inventoried and tiered by risk level
• Pre-deployment validation of AI models includes security assessment appropriate to the model's risk tier
• Ongoing monitoring covers security-relevant performance dimensions, not only accuracy and stability
• Adverse outcomes from AI systems, including those that may indicate adversarial attack, are reported through the model risk reporting chain
• The board and risk committee receive reporting on AI model risk that includes security dimensions

We work with financial institutions to assess where AI security sits within their existing model risk management framework and to develop the assessment and monitoring capabilities that close identified gaps. Contact us at info@cyberlinx.com.au to discuss how we approach this work.