30 July 2012

SMotW #17: audit findings

Security Metric of the Week #17:  number and severity of audit findings 

Our latest 'security metric of the week' builds on the following premises.  

Firstly, the number and severity of audit findings bears some relationship to the state or maturity of the organization's governance, risk, compliance and security arrangements, along with the number, quality, scope and depth of the audits.

Secondly, since audits are invariably independent and formal, the number of audit findings is an objective, cheap and easy-to-obtain measure, as is the 'severity' (or gravity or importance) provided findings are routinely rated/classified by the auditors, which they usually are.  The severity of audit findings also helps focus management attention on the issues that really matter.

[We are of course assuming that "audit finding" is a recognized term.  Most if not all audit functions generate reports that identify and discuss discrete findings.  Many also explicitly identify "audit recommendations", again as discrete items in the reports, so counting them is also a possibility.

This metric may be presented in purely numeric form (e.g. as graphs or pie charts or whatever), as text (e.g. a tabular report outlining each of the findings along with other relevant information such as when it was raised, when it should be actioned and closed, and who is responsible for the action) or both (annotated numbers or graphics).

When designing and specifying the metric, management probably ought to decide whether it takes account of the findings from internal, external and certification audits, management reviews and/or risk assessments etc., although it may not be necessary to define this formally: it could perhaps be managed dynamically according to the nature and number of issues to be reported (e.g. ignoring the less important findings/recommendations to concentrate on the biggies, whatever their source).

The metric may be a useful high-level/strategic metric, particularly as it is highly Independent and hence a Genuine measure, unlikely to be substantially manipulated by someone gaming the system.


Notice that, as worded above, the metric is not about information security findings specifically: all findings are counted.  You may think it better to distinguish those audit findings that specifically relate to  security, but doing so begs questions about how findings are categorized.  Perhaps the auditors can be persuaded to categorize their own findings?  It could be argued that practically everything in audit reports relates to information security in some fashion, and at the end of the day, management is not solely concerned with information security so does it really matter anyway?

With hindsight, the PRAGMATIC score of 77% that we calculated for the metric is probably on the low side: it looks as if we were rather pessimistic on the cost factor, especially if audit already creates and uses/reports the raw data for other purposes i.e. on their reports and in their databases used to track audit findings and recommendations.  [By the way, there are probably other sexy numbers and information in audit's databases that could be used for further security metrics, provided they are not so confidential that they cannot be shared!] 

27 July 2012

Security awareness/training metrics

An interesting discussion on the ISO27k Forum concerns measuring security awareness and training activities.  Most of the measures proposed so far have been 'input' or 'process' metrics such as evaluation sheets measuring things such as the quality of the venue, the course materials, the food served, the tutor (even the parking spaces!).  Many organizations collect basic data such as the number of attendees at awareness and training events, and the number of events attended by each employee in the course of a year.

Input measurements of this nature are relatively cheap and easy to collect, in volumes large enough for meaningful statistical analysis, and some of those statistics are actually useful for management purposes (e.g. distinguishing good from bad trainers, and identifying areas for improvement in their techniques or the materials or the venue).  A few may even be required for compliance reporting purposes against [senseless] regulatory requirements such as "All employees must go through security training once a year" (which says absolutely nothing about making the training effective!). 

However, 'output' or 'outcome' metrics tend to be much more valuable, although it takes some creativity to think up metrics that measure the actual effects of security awareness and training.  Examples include:
  • Comprehension tests to assess how well employees recall and understand key lessons
  • Behavioral surveys to measure the organization's security culture
  • Phishing, social engineering and penetration tests etc. to determine how well employees recognize and respond to mock attacks
  • Audits and post-incident reviews of actual incidents (including noncompliance) to determine the extent to which lack of training/awareness was a factor
  • 'Security maturity' metrics that rate the organization's awareness and training practices against generally accepted good practices [see an example of this kind of metric].
Both input and output metrics should show the effects of changes in the awareness and training practices (e.g. running a "fraud focus month" with a bunch of extra activities on that topic), however output metrics measured over successive months are much more likely to demonstrate whether the effects were long-lasting and had the desired effects.

The PRAGMATIC method deliberately emphasizes the value of metrics that are Relevant, Predictive, Meaningful and Cost-effective: naturally cost is a factor since audits and surveys can be quite expensive, but that's not the whole story: the additional value of output metrics in supporting key management decisions compared to  cheap-n-nasty input metrics means they are often an excellent investment.



PS  Penny Riordan pointed out the Kirkpatrick model for evaluating training. Professor Kirkpatrick elaborated on four 'levels' or methods or evaluation as follows:

  1. Reaction of student - measure what student thought and felt about the training;
  2. Learning - measure (by testing) the resulting increase in knowledge or capability;
  3. Behaviour - measure (by observation) the extent of behaviour and capability improvement and implementation/application;
  4. Results - measure the effects on the business or environment resulting from the trainee's performance.
The third/fourth levels correspond to most of what I called 'output' or 'outcome' metrics, while the second level corresponds with the comprehension tests I mentioned.  Training feedback forms equate to level 1, part of what I called 'input' measures.

Thanks Penny!

23 July 2012

SMotW #16: policy noncompliance

Security Metric of the Week #16: number of security policy noncompliance infractions detected

The extent to which employees comply with the organization's security policies sounds like the kind of thing that management might want to track and, where appropriate, improve.  This week's metric is a typical, if rather naive attempt to measure policy compliance ... by counting noncompliance incidents.

Policies are 'mandated', in other words management expects everyone to comply with them unless there are justified reasons not to comply (meaning authorized exemptions for those organizations that are mature enough to appreciate the need to manage this aspect carefully).  While management originates most of the security requirements documented in policies, some derive from external obligations under applicable laws, regulations or agreements with third parties (e.g. PCI-DSS).  

The metric's wording implies that unauthorized noncompliance 'infractions' (more often called incidents) are being detected and recorded in a form that can be counted - typically some sort of security incident database, usually part of a Help Desk ticket management system.  Why not simply report the number of security incidents recorded by the Help Desk over, say, a month?  Such a metric would be low cost, but what about its benefits?

In reality, many other noncompliance situations occur, and some of them are detected or identified but for various reasons don't get reported as incidents.  As an example, who would bother reporting an everyday tailgating incident, even in a military organization that prides itself on physical security?  Furthermore, lots more noncompliance incidents are not even identified as such - they are not observed or recognized, or they are very short-lived or otherwise deemed trivial.  If an employee spots and challenges a tailgater, who then proceeds to identify themselves with an authentic-looking staff pass, the 'incident' is such a non-event that it is most unlikely to be reported, but it could of course be an actual intrusion.

All of this constitutes a huge bias to the metric as worded, a tremendous source of random error or noise in the measurement values.

Maybe it would help if we clarified the metric by reporting not the absolute number of policy noncompliance incidents but the rate of occurrence i.e. the number of incidents in a predefined period.  What would it mean if the metric jumped from 97 to 145 one month?  Is that something that should concern management?  What if it went from 145 to 97, or from 97 to 99, or hit zero?  How, exactly, would this metric support the decision making process?  Precisely what would management be expected to do with it?

Probing questions of this nature soon belie this metric's superficial allure.  It is not hard to find fault with it.  Arguably the most fundamental issue is that it is practically impossible to determine the true number of noncompliance incidents by direct observation, except perhaps in strictly controlled experimental conditions.  The best we can reasonably hope achieve in reality is to estimate the true number as rationally and accurately as we can, for instance by statistical means, using a more scientific process for identifying and reporting noncompliance incidents, such as periodic security policy compliance audits.  Unfortunately, that approach substantially drives up the Cost of the metric and so adversely affects its PRAGMATIC score:


We have been quite generous on the 75% rating for Actionability on the assumption that, if the measurements were poor, management would initiate whatever they considered appropriate to improve policy compliance, such as training and awareness activities coupled with increased management oversight, and perhaps more emphasis on enforcement actions.  We didn't have the same latitude with the rating for Accuracy, although using auditors or other professional assessors to measure the metric could improve its Independence, relative to self-reporting of noncompliance incidents by employees.

The Genuineness rating suffers largely because, if this metric were being reported and used proactively by management in an attempt to improve policy compliance, there is a distinct possibility that it would be deliberately manipulated.  There are some obvious if crude ways in which employees might 'game the system' to drive up the metric without materially improving compliance, such as consciously failing to report noncompliance incidents.  Even if management succeeded in addressing these tricks (e.g. by instituting and enforcing a policy on reporting incidents), other more subtle games would probably flourish.  It is amazing how creative people can get in the face of adversity!

The Predictability rating is also depressed because it is a backwards-looking metric: it tells us how things were in the preceding period but doesn't say much about how they may change in the forthcoming period, other than vague indications that might emerge from the background noise.

It is a reasonable assumption that security policies themselves are directly Relevant to information security, hence compliance with the policies is also Relevant.  However, there is more to security than policy compliance (it is 'necessary but not sufficient'), and as noted elsewhere the metric as worded does not perfectly reflect policy compliance, hence we rated the metric 64% on the Relevance criterion.  [This paragraph illustrates the PRAGMATIC thinking behind each of the ratings.  If we had the time and energy, we should probably document all the ratings on all the metrics for future reference, but in practice we would be more inclined to elaborate on and write-up the rationales for the few security metrics that we eventually adopt.]

The overall PRAGMATIC score of 57% tells us that, as originally stated, this is unlikely to feature as one of the few good security metrics we would chose, unless perhaps we were so short of inspiration that there were no higher-scoring candidate metrics on the table.  

Having discussed our concerns and hinted at some of the ways in which we might improve this metric, do you have some even better suggestions?  Or do you agree that this metric is essentially doomed?  Submit a comment and we'll do our best to respond positively.

20 July 2012


Hitherto - before the PRAGMATIC method was invented - deciding which security metrics to measure was a black art, a highly subjective decision making process.  One might even question whether organizations actually 'select' security metrics deliberately, systematically and rationally.  

Think about that for a moment.  Why does your organization measure whatever it does measure in relation to information security?  Does that mean that management doesn't care about all the other security stuff you could also measure?  Does it really matter what security metrics you use?

Pre-PRAGMATIC organizations presumably measure certain facets of information security because: 
  • They are cheap and easy to report, typically because the raw numbers are readily available (some systems generate pretty graphs straight out of the box, but does management need them?);
  • They are recommended by someone, peers claim to measure them, or the organization is required to report them by some third party (e.g. an authority such as a regulatory body or owner);
  • 'It's the way we've always done it' ... (brains in neutral!);
  • 'It seemed like a good idea at the time' ... although we may no longer recall why, and things have probably changed in the interim; 
  • 'It's obvious!' ... but probe deeper and you may discover a mix of rational and irrational reasons for focusing attention on those particular aspects, and often a lack of appreciation of the opportunity cost i.e. it might be better to measure other things, or measure the same things in other ways;
  • Management think they couldn't possibly measure the security things they really care about, and hence are forced to accept whatever metrics they are offered (nonsense!  Read Douglas Hubbard's book and get creative!).
Once an organization adopts the PRAGMATIC approach, selecting security metrics becomes a much more straightforward process.  The chosen metrics can be described and justified lucidly and convincingly, giving legitimate reasons for choosing them.  Other potential metrics can be assessed objectively in relation to the existing metrics suite, using the PRAGMATIC criteria as a framework for the decision-making process.  The PRAGMATIC approach lets management identify and weed-out low-scoring security metrics that aren't earning their keep, not just saving the associated metrics generation and reporting costs but focusing finite management attention on more valuable metrics.

16 July 2012

SMotW #15: HR security maturity

Security Metric of the Week #15: Human Resources security maturity

In order to explain the PRAGMATIC score for this week's example security metric, we first need to introduce you to the concept of security maturity metrics.  Bear with us.

Section 8 of ISO/IEC 27002:2005 lays out a suite of HR-based information security controls that apply to the pre-, para- and post-employment phases.  For example, prior to offering anyone a role, especially in a powerful/trusted position, wise organizations conduct suitable background checks to weed-out unsuitable candidates.  For highly sensitive government and military work, the security clearance process can involve an extensive range of checks including credit worthiness, criminal history, identity, qualifications, professional experience and character references, in addition to a structured interview process and on-the-job supervision/oversight during a probationary period.  For low-grade positions such as office cleaners, pre-employment checks tend to be trivial and in many cases are left to third parties supplying contract staff ... but while this is common practice, it creates information security risks that probably deserve more attention.  Office cleaners typically work alone out-of-hours and have ready access to IT equipment, paperwork and other valuables:   do you really want to let someone of unknown vintage and negligible loyalty, paid a minimal wage, loose in your office?

So-called maturity metrics are an excellent way to measure such complex situations.  The idea is simply to lay out a spectrum of good practice security controls ranging from trivial, negligible and weak up to extensive, leading-edge and strong.  Figuring out where an organization stands in relation to the spectrum of controls allows us to determine a maturity level or score. 

Maturity metrics are similar to the Capability Maturity Models originally created [and trademarked] by Carnegie Mellon University as a means to assess and measure software development practices.

Given the number of aspects relevant to information security, over the course of about two decades we have developed a tabular style maturity metric with rows for each type or area of control and columns for key points on the scale.  Cells in the body of the table contain examples of the controls, providing sufficient information to guide a competent reviewer in determining the most appropriate level and hence the score.

While scoring process is inevitably subjective, stating the scoring criteria makes it objective enough that different reviewers, auditors, managers, assessors etc. can gather, consider and discuss the evidence, generally reaching agreement on specific maturity scores.  

We have provided a suite of security maturity metrics based on the recommendations in ISO/IEC 27001 and 27002 as an appendix to our book.  As an example, here is one of the seven rows in the table covering section 8's HR security controls:

No human resources security
Basic human resources security
Good human resources security
Excellent human resources security
Information security rôles and responsibilities are entirely undocumented
Some information security rôles and responsibilities are documented, though not very well or consistently
Most information security rôles and responsibilities, including all the important ones, are assigned to individuals through being incorporated into vacancy notices, job descriptions and/or codes of conduct
Information security rôles and responsibilities are comprehensively documented, formally assigned to suitable individuals (typically in legally-binding contracts of employment or terms and conditions of employment), and are  proactively maintained
(e.g. periodically reconfirmed with the individual’s signature to confirm their acceptance)

The four columns correspond to maturity scores of 0%, 33%, 67% and 100% respectively.  There is a further implied scoring point at 50%, marking the divide between practices that are generally considered unacceptable and those that are broadly acceptable.

OK, so now let's look at the PRAGMATIC rating for this kind of metric:


The metric scores remarkably well, especially given that, while HR practices are undoubtedly relevant to information security, they are rather difficult to measure.  In fact, to our knowledge, very few organizations have decent security metrics in this area.  Most seem content with classical HR metrics such as the number of employees that have completed some form of security training, despite the fact that such metrics do not adequately reflect security in practice.  Did anyone actually learn anything from the training?  Did they actually change their behaviors as a consequence?  Or did they just turn up under sufferance and passively sit there just to get the tick on their personnel records?  The classical metric says practically nothing about these aspects.  [Feel free to determine the PRAGMATIC score for the classic metric as a homework exercise.  We'd be amazed if it scores anything remotely approaching 86%!]

There are further security maturity metrics in the book, corresponding to the remaining sections of ISO/IEC 27001 and 27002.  We'll expand on them in future metrics of the week but for now you will have to bide your time until the book is published, although we will try to address any questions in the comments to this blog.

09 July 2012

PRAGMATIC Security Metric of the Quarter #1

PRAGMATIC Security Metric of the First Quarter

Having scored and discussed fourteen Security Metrics of the Week during the first three months of this blog, it seems appropriate now to take a step back, review the metrics we have discussed thus far and consider the value of the PRAGMATIC process.  

Here are the fourteen security metrics, tabulated in descending order of their overall PRAGMATIC scores.  Click any of the metrics to read more about them and discover why we scored them thus.












Discrepancies between physical location and logical access location

75 76 72 90 82 75 85 83 60 78%
Information security policy coverage

75 82 92 78 80 70 73 60 81 77%
Unowned information asset days

40 51 84 77 74 86 92 94 82 76%
Number of unsecured access points

95 80 90 70 85 77 45 75 55 75%
% of critical controls consistent with controls policy

83 92 80 83 89 82 32 70 35 72%
Days since logical access control matrices for application systems were last reviewed

55 80 95 30 80 85 60 70 80 71%
Number of unpatched technical vulnerabilities

80 64 80 70 80 75 25 85 52 68%
Coupling index

68 85 50 60 72 47 35 61 42 58%
Vulnerability index

74 85 71 74 60 32 46 33 19 55%
System accounts-to-employees ratio

74 67 38 39 68 42 36 83 44 55%
Corporate security culture

80 80 60 55 75 55 10 45 10 52%
% of purchased software that is unauthorized

71 51 90 75 82 35 13 20 6 49%
Security budget as % of IT budget or turnover

13 3 16 2 2 0 4 18 88 16%
Number of firewall rules changed 2 1 1 10 2 33 14 4 17 9%

In simple numerical terms, the metric Discrepancies between physical location and logical access location is the leader of this little pack which qualifies it as <cue drum roll> our first PRAGMATIC Security Metric of the Quarter.  In fact there's clearly not much to choose between the top four metrics in the table in terms of their overall PRAGMATIC scores.  The scores and hence rankings may well have changed if we had made different assumptions in the scoring, or of course if we had altered the specification/wording of individual metrics to address the issues we identified and hence altered their scores.  Furthermore, your scoring of the metrics may differ from ours due to  differences in how we each understand and interpret both the metrics and the PRAGMATIC approach.  We don't have the same experience as you, our biases differ, our presumptions and organizational contexts differ and no doubt we have different potential audiences and purposes in mind for these metrics.

That whole line of discussion is moot, however, since we are not claiming that the PRAGMATIC approach is scientific and objective.  Our top-scoring metric is not necessarily the best of the bunch under all circumstances for all organizations, just as the lowest scoring metric may be appropriate and score more highly in certain situations.  The approach simply offers a rational way to consider the value of and compare various security metrics, to elaborate on their pros and cons, to identify ways in which the metrics might be re-phrased or materially altered to improve them, and most of all to facilitate a more informed and productive metrics discussion with management.  Even if you simply use the process to shortlist the most promising from a bunch of metrics candidates, leaving the final selection to management, isn't that a worthwhile outcome?

There's plenty more to say yet about being PRAGMATIC, including ways to glean further useful information from the data in the scoring table above, but we'll leave that for the book, future blog pieces, seminars and papers.   Meanwhile, do please let us know about your favorite security metric and we'll see we make of it.  We look forward to your comments on this blog and emails, especially constructive criticism and creative ideas to make the PRAGMATIC approach even more effective.  Over.