How to Measure Whether Security Awareness Training Is Actually Working

The most common metrics used to evaluate security awareness training are completion rates and phishing simulation click rates. These are reported to boards and audit committees as evidence that the training programme is working. They are not. Completion rates tell you that people sat through the training. They tell you nothing about what was retained, what changed, or whether the organisation is more secure as a result. Phishing simulation click rates tell you how many people clicked a simulated email. They are a narrow measure of a single skill, and they are vulnerable to gaming, fatigue, and other confounds that reduce their reliability as indicators of genuine capability.

The gap between these metrics and actual programme effectiveness is significant, and it matters because organisations make investment decisions based on what they measure. If a programme shows declining click rates and high completion percentages, it looks like it is working. If those improvements do not correspond to any change in real-world security behaviour, the investment is producing reporting value rather than security value. Measuring more meaningful outcomes is not difficult, but it requires deciding in advance what outcomes the programme is designed to produce.

Incident Reporting Rate as a Behaviour Indicator

Incident reporting rate is one of the most useful indicators of genuine awareness programme effectiveness. It measures whether staff are applying their training in the real world: noticing suspicious activity and doing something about it. A programme that produces no change in incident reporting rates over time has not changed the habits that matter most. A programme that produces a sustained increase in staff-initiated reports, even as email filtering improves, is demonstrating real behaviour change.

Tracking reporting rates requires a reliable reporting mechanism and consistent categorisation of what is being reported. Many organisations already have a mechanism but do not systematically analyse reporting patterns over time or connect them to training activity. Segmenting reporting data by role group, department, or time relative to training events reveals which parts of the programme are producing changed behaviour and which are not. A finance team whose reporting rate increases significantly after a targeted training session is providing direct evidence that the training produced the intended outcome.

Knowledge Retention Testing

Training that produces short-term knowledge gain without retention is not producing lasting behaviour change. Testing knowledge retention at intervals after training, rather than only immediately following it, gives a significantly more accurate picture of what has actually been learned. A staff member who scores well on a quiz immediately after completing a module and poorly on the same questions three months later has not retained the learning. The programme design needs to address this with spaced reinforcement and retrieval practice.

Retention testing should also test for application, not just recall. Asking someone to define phishing is less useful than presenting them with a scenario and asking what they would do. Scenario-based assessment tasks are more predictive of real-world behaviour than knowledge quizzes because they test the decision-making process, not just the vocabulary. Building these assessments into the programme at regular intervals (not just at the end of a module) provides much more useful data about whether the training is producing the right cognitive habits.

Near-Miss Analysis and Simulation Realism

Near-miss reporting, where staff report an action they almost took before realising it was suspicious, is a particularly useful signal. It indicates that the recognition process worked in a real-world context, even if only at the point of hesitation. Organisations that actively encourage near-miss reporting and treat those reports as valuable information rather than admissions of near-failure tend to see a clearer picture of how well their training is transferring to real-world decision-making.

Simulation realism is a related consideration. If phishing simulations are becoming widely recognised as simulations (because staff have developed pattern recognition for the way the organisation runs them, rather than for phishing itself), click rates will decline without any real improvement in security awareness. Testing this by varying simulation design and tracking whether declining click rates coincide with increasing reports of the simulation itself (rather than reports that treat it as a genuine suspicious email) reveals whether the measure is valid. A staff member who correctly identifies a simulation email and reports it as suspicious is demonstrating the right behaviour. A staff member who recognises it as a simulation and ignores it is not.

Trend Analysis Over Time

Single-point measurements are less useful than trends. A programme that is producing genuine behaviour change should show improving trends across multiple indicators over time: increasing incident reporting rates, improving retention test scores, decreasing time-to-report for suspicious activity, and declining rates of security policy exceptions or bypass requests. No single metric is definitive. A portfolio of consistent trends across multiple measures is much stronger evidence that the programme is working.

• Completion rates and click rates measure activity, not behaviour change.
• Incident reporting rate is a strong real-world indicator of awareness programme effectiveness.
• Knowledge retention testing at intervals after training reveals what has actually been retained.
• Scenario-based assessments are more predictive of real-world behaviour than knowledge quizzes.
• Near-miss reporting provides direct evidence of training transferring to real decisions.
• Trend analysis across multiple indicators is more reliable than any single metric.

Cyberlinx designs awareness programmes with measurement frameworks built in from the start. If you want to know whether your training is actually producing behaviour change, contact us at info@cyberlinx.com.au.