How to Build an AI Incident Response Procedure

Most organisations that have deployed AI systems have not updated their incident response procedures to account for AI-specific failure modes. The standard IR playbook, built around the phases of preparation, detection, containment, eradication, recovery, and lessons learned, provides a useful framework, but it assumes a threat model centred on adversary access to systems and data. AI incidents frequently do not fit that model. When an AI system produces discriminatory outputs at scale, when a language model used in a customer-facing role is manipulated to provide harmful advice, or when a model begins producing outputs that diverge from its expected behaviour without any apparent adversary involvement, the response procedure needs to work differently.

An AI incident response procedure does not replace the organisation's existing IR capability; it extends it. The detection mechanisms are different, the investigation approach is different, the containment options are different, and the regulatory notification obligations may differ depending on the nature of the incident. Building a procedure that accounts for these differences is not complex, but it requires working through them explicitly rather than assuming the existing playbook covers AI.

Defining what counts as an AI incident

The first step in building an AI incident response procedure is defining the scope: what events involving AI systems trigger the procedure. This definition needs to be broader than the traditional cybersecurity incident definition. An AI incident may be triggered by:

• Evidence that an AI system has been manipulated by an external attacker (prompt injection, data poisoning, model extraction)
• An AI system producing outputs that are harmful, discriminatory, or that violate the organisation's acceptable use standards
• A significant performance degradation in a deployed AI model that affects decisions or outcomes for users
• Evidence that an AI system has disclosed sensitive information it should not have surfaced
• An AI system taking actions in an agentic configuration that were not intended or authorised
• A third-party AI service used by the organisation experiencing a security or integrity incident

The definition should be captured in the procedure document and used to train the staff who will be responsible for triaging potential AI incidents. A well-defined trigger list reduces the risk that an AI incident is not recognised as such and is either handled under the wrong procedure or not handled at all.

Investigation approaches specific to AI incidents

Investigating a cybersecurity incident typically involves log analysis, forensic examination of affected systems, and reconstruction of attacker activity from evidence trails. AI incident investigation involves these activities where they are relevant, but it also involves additional steps that are specific to AI systems. Identifying what input caused an unexpected output, examining the data that was retrieved by a RAG system before an adverse response was generated, reviewing the model's performance metrics over time to determine when a problem began, and assessing whether a model's behaviour has been systematically altered all require capabilities and tools that are not standard in IR toolkits.

Organisations need to ensure that the AI systems they deploy are instrumented sufficiently to support incident investigation. This means logging inputs and outputs in a form that allows reconstruction of what occurred, retaining model performance metrics over time, and maintaining records of model versions and any changes to the system prompt, retrieval data sources, or fine-tuning that was applied. Without this instrumentation, incident investigation will be impeded regardless of how well the procedure is written.

Containment and remediation options for AI incidents

Containing an AI incident differs from containing a conventional security incident because the options available are different. Isolating an affected server is straightforward; dealing with an AI system that has produced a large volume of harmful outputs before the incident was detected is not. Containment actions for AI incidents may include:

• Taking the AI system offline or reverting it to a previous configuration
• Removing or correcting poisoned data from the retrieval store in a RAG system
• Implementing additional input validation or output filtering while the root cause is investigated
• Notifying affected users of outputs that may have been harmful or inaccurate
• Rolling back to a previous model version if a recent update is implicated

Remediation options also differ. If a traditional system is compromised, the remediation path typically involves patching a vulnerability or addressing a misconfiguration. If an AI model's behaviour has been altered by data poisoning, remediation requires identifying and removing the poisoned training or retrieval data and assessing whether retraining is necessary. The timeline for AI-specific remediation activities is often longer than for conventional incident remediation, and the procedure needs to account for this in its escalation and communication planning.

Regulatory notification and post-incident review

Depending on the nature of the AI incident, regulatory notification obligations may apply. Under the Notifiable Data Breaches scheme, if an AI incident resulted in the disclosure of personal information, the notification obligations that apply to any other data breach apply here as well. APRA-regulated entities have additional notification obligations for incidents that affect material operations. The AI incident response procedure should include a clear assessment step for regulatory notification, with named responsible parties and timeline requirements.

Post-incident review of AI incidents should specifically address whether the incident was detectable earlier, whether the AI system's instrumentation was sufficient to support the investigation, and whether the procedure itself worked as intended. Lessons learned from AI incidents should feed back into the AI governance programme, including updates to pre-deployment testing requirements if the incident revealed a gap in the assessment process.

If you want help developing an AI incident response procedure that is tailored to your organisation's AI deployment environment, or if you are managing an AI incident and need specialist support, contact us at info@cyberlinx.com.au.