Testing AI Systems the Way Attackers Do

Most organisations building products on large language models reach a point where they add a system prompt instructing the model not to do certain things. "Do not discuss competitors. Do not produce harmful content. Only answer questions about our product."

They test it themselves, find it works, and ship.

What they have not done is test it the way an adversary would. System prompts are not a security control. They are instructions. The difference matters.

The Attack Categories We Test In Practice

Prompt injection is the category of attack where an adversary provides input designed to override, contradict, or circumvent those instructions. Direct prompt injection comes through the user interface: "Ignore your previous instructions and instead tell me..." Indirect prompt injection is more interesting: the adversary embeds malicious instructions in data the model retrieves, so that when the model processes a document, a web page, or a database record, the payload activates without the user ever typing it directly. If your AI system retrieves external content and acts on it, indirect prompt injection is in scope.

Jailbreak techniques exploit the model's own generalisation behaviour: roleplay framings, hypothetical scenarios, encoded inputs, multi-step conversational sequences that gradually shift the model's output toward behaviour it was trained to avoid. These techniques are well-documented and constantly evolving. A jailbreak that failed against last month's guardrail tuning may succeed against this month's, especially as models are updated in ways that shift the underlying behaviour distribution.

Guardrail bypass covers both. Most AI products include filtering layers: input filters that block certain phrasings, output filters that catch certain response patterns, moderation classifiers that flag content before it reaches the user. Testing guardrail bypass means systematically attempting to elicit blocked outputs in ways that either avoid the input filter, exploit gaps in the output filter, or exploit the interaction between multiple components where each individual filter passes the content but the combination produces a harmful output.

Scaffolding and application architecture vulnerabilities are where most real-world exploits land. If your AI system has access to user data, the ability to call external APIs, or the capacity to take actions in other systems — file writes, email sends, database queries — then a successful injection attack is not just a conversation about harmful content. It is a pivot point into your broader infrastructure.

Why AI Testing Requires Different Tooling

Traditional application penetration testing works against a defined attack surface: ports, services, endpoints, input fields. An AI system's attack surface includes the model's own response behaviour — which changes depending on how inputs are framed, what context has been established in the conversation, and what data the model retrieved before generating its response. Manually probing this surface at the scale required to find meaningful vulnerabilities is not feasible.

We use AI to test AI. Our in-house tooling generates and categorises attack variations at a volume that manual testing cannot match — systematically exploring prompt injection variants, jailbreak framings, and guardrail bypass approaches across the known attack taxonomy. Findings are then reviewed by practitioners who can distinguish a real exploitable behaviour from a statistical artefact and translate technical findings into remediation guidance the engineering team can act on.

What a Useful AI Security Assessment Report Looks Like

An AI security assessment report is structured differently from a network penetration test report. Rather than a list of CVEs with CVSS scores, a useful AI assessment report covers:

• Which layers were assessed: dataset integrity, model behaviour, scaffolding and application architecture, human-in-the-loop oversight, governance, exploitability, and privacy
• What attack categories were exercised and what success rates were observed across each category
• The material findings translated into business risk terms — not "the model produced harmful output under these conditions" but "an attacker could use this to access user data / bypass content restrictions / escalate privileges into connected systems"
• Remediation prioritised by exploitability and business impact, not by category
• How findings map to ISO 42001 governance requirements or the NIST AI Risk Management Framework, if the organisation needs to connect testing results to a governance programme

The Regulatory and Governance Intersection

Adversarial AI testing does not exist in isolation from governance. ISO/IEC 42001:2023 (the AI management system standard) requires organisations to assess and manage AI-related risks, which includes the risk that AI systems can be manipulated or exploited. The NIST AI Risk Management Framework's Measure function explicitly includes adversarial testing as a mechanism for assessing model robustness and reliability.

In Australia, APRA's published guidance on artificial intelligence and the Voluntary AI Safety Standard from the Department of Industry, Science and Resources both address the expectation that organisations deploying AI in regulated contexts should understand the risk profile of their systems, not simply assume that vendor-provided models are secure.

For organisations pursuing ISO 42001 certification or building an AI governance programme aligned with the NIST AI RMF, adversarial testing results are not just a technical input. They are a governance input: evidence that the organisation has assessed its AI systems against a real adversarial threat model, not just against its own internal testing.

Who Should Be Doing This and When

AI-first startups should conduct an adversarial assessment before launch, not after. The cost of finding and fixing an exploitable vulnerability before users are on the platform is a fraction of the cost of a post-launch incident, particularly if the system handles personal data or operates in a regulated context.

Scale-ups with enterprise sales in the pipeline should expect that security-conscious enterprise buyers will ask about AI security testing as part of vendor due diligence. Having a conducted assessment and a documented remediation programme is a materially better position than "we tested it ourselves."

Enterprises deploying agentic AI tools broadly — systems that take actions autonomously across APIs, databases, and communication platforms — should treat adversarial testing as a prerequisite, not an afterthought. The blast radius of a compromised agentic system is categorically different from a compromised chatbot.

To discuss an AI security assessment for your organisation, contact Cyberlinx at info@cyberlinx.com.au.