Build a Security Culture (2015)
CHAPTER 6: MEASURING CULTURE
In this chapter I will look at a few ways to measure culture, and how you can take existing data to use as a baseline.
One thing I often hear from fellow security professionals is that it is impossible to measure awareness and culture. It is an interesting point of view, and one that is usually based upon:
• not knowing how to measure soft skills.
• previous failures to create results from awareness activities.
It often boils down to not realising that awareness and culture are reflected in the behaviours of employees. In most organisations today, the heavy use of computer systems enables us to closely monitor any and all use. An example:
Bob, a salesperson who has been in your organisation for two years in March, uses a combination of a laptop, a pad and his smartphone to interact with the computer systems. He reads his emails, he uses customer relationship management software, he registers his expenses and so on. All of these systems are set up to log every interaction with users, John’s included. The purpose of the logging is to ensure high-quality service, backtrack activities to see if there was something a particular user did to cause problems, and receive early warnings on potentially disastrous changes in the computer systems.
Every time Bob uses the systems, his data is being recorded: timestamped, geolocated, device used, system used and so on. These logs show Bob’s current behaviours, including his habit of eating his lunch at a café down the road. Of course, you do not know that he is eating his lunch there, what you know is that his device, using his credentials, is being used almost daily to connect from that location.
What you see is how Bob is using the computer systems. That is what social scientists call behaviour. Bob is interacting with his surroundings, and you are logging that interaction. You are measuring his behaviour.
Bob’s employer has a bring your own device (BYOD) policy, stating that connection to the computer systems must only occur using a virtual private network, and the use of public Wi-Fi is not allowed. When on the road, Bob should only rely on his mobile Internet connection.
When you examine the connection logs for the sales team, you discover that most of John’s colleagues fail to follow the policy. Instead of using their mobile network, they prefer to connect from Wi-Fi networks on the road.
Using the current log data, you now have a baseline behaviour. You know your current situation, the “as is”.
The baseline measurement is important when designing change: knowing where you are makes it possible to navigate to the location you need to be. If you know where you want to be, that is.
In John’s organisation, the goal state is described in the policy document:
Every worker outside our premises should connect using VPN, and only through mobile networks or previously accepted networks.
In this case, defining your goal is quite easy: you need employees to connect to the computer systems with VPN, and from pre-defined networks only. You also know that Bob and his colleagues are far from this goal; their behaviour is not according to your goal.
The baseline measurement shows a clear gap from your defined goal. That gap is what you will bridge with your security culture programme. Using the Security Culture Framework, you are now ready to take a closer look at Bob and his colleagues, analysing their behaviour, their current security understanding and their preferred communication style. Next, you choose activities that will resonate with Bob, helping him understand why he needs to follow the company policy.
Part of your analysis should be to interview Bob, possibly even going out on the road with him for a day, so you can better understand the situation from his perspective. He might tell you that the mobile network is so slow that it makes it impossible to do his job. Maybe he is unaware of his phone automatically connecting to open Wi-Fi networks, There are a number of possible explanations for Bob’s current behaviour, and understanding his side will provide you with a better idea of how to change it.
After creating your baseline and analysing Bob’s current behaviour, you create a security culture campaign, using the Security Culture Framework, with a selection of different activities, all directly related to Bob and his team’s needs. Your programme consists of a six-week nano-learning programme with two five-minute video clips each week. You also join two sales meetings, one at the start of the programme and one towards the end, where you use a Pineapple device to demonstrate what a man-in-the-middle attack may look like, surprising everyone attending at just how easy it is to intercept traffic. You also distribute a keyring with the text “Lock My Door”. Before you start the programme, you run a short five-question survey where employees are asked about their general security knowledge. You rerun the same survey a week after your programme finishes, giving you another source of information that you can correlate with your logs.
After your security culture campaign has come to an end, you take another look at your logs where you discover a steady change in how the sales department connect to the main systems. Most of the sales force now connect using VPN and the mobile networks while on the road.
You also notice that Bob no longer seems to have his lunch at that diner. Looking at the geolocation data your logs collect, you notice he is connecting from one of the subsidiaries, where he gets access to a high-speed Internet connection without breaking the company policy.
Comparing the results from your two runs of the survey, you also notice a clear change in the security knowledge and understanding of the sales team. They show a clear trend towards understanding why the policies are in place, and that even though the policies demand a behaviour that to the salespeople seems counterproductive (getting in the way of their work), they now realise that if they fail to follow the policies, they effectively put the workplace in danger.
The argument that it is very hard to measure awareness and behaviour change is flawed26. Just like anything else, it comes down to your current knowledge and skill-set. If you have never learned how to look for behaviour data in your systems, you are not to blame. I hope that reading this chapter has spawned a few ideas as to how you can use your own data sources to look for behavioural data.
Using your current data sources is a great way to look at how behaviour translates into patterns in your logs. There is usually no need to buy another system or software to create more data when you want to measure awareness and behaviour change. Most organisations I work with have more than enough data points and logs are readily available. Sometimes you need to turn on a logging feature in the system. Most of the time, however, the challenge is to select just the right data to use from the abundance of available data.
Another challenge that sometimes arises is the need to do proper analysis on the data. Not every security professional is also a skilled data analyst. Most security professionals are not social scientists. We, the security professionals, tend to come from a hard-science background, where only what we see directly is considered an acceptable truth.
When it comes to understanding people, behaviour, awareness and culture, we need to learn from the social scientists. There are a number of scientific tools and methods used by psychologists, sociologists and anthropologists around the world. These include both quantitative data (think of your logs) and qualitative data (think of an awareness survey or maturity model). Arguments still rage about which one of these methods is best when understanding people: the current consensus is that we need both methods to create a more complete understanding of how we behave and how change can be controlled.
The understanding that we may need both quantitative and qualitative data to create a wider understanding of behaviour and change is important to notice in security culture. Your logs tell so much (or so little), and surveys are biased by both the questions and their wording, as much as by the context and understanding of the participants. Understanding that both quantitative data and qualitative data can lead you far away from your path will help you look for ways to assure the quality of the data, and its validity to the real world, as quickly and early as possible. Correlating quantitative data with qualitative data may help you discover discrepancies and problems with your hypothesis.
Where do you look for data?
You can use a number of different sources for information on behaviour - either directly or indirectly. Your budgets may impose limits (as they should), and so may your own interest and skill-set. A data analyst is a great asset to any security team, and can identify relevant data sources as well as creating the necessary analysis.
In the book Data-Driven Security, Jay Jacobs and Bob Rudis walk the reader through how to set up and run your own security analytics using R and Python. In their dashboard chapter, they include an example of a CISO dashboard on security awareness based on the SANS awareness maturity survey. I strongly suggest reading that book, even if you are not a data analyst. Their clear explanations based on real security issues make it very easy to relate to the topic, making it fun to learn!
Most computer systems today come with immense logging opportunities, giving you vast amounts of data to analyse your employees’ behaviours on your systems. Often, all it takes is knowing where to look, and turning the logging on.
Surveys can provide a lot of information. There are also a number of challenges with surveys, including the fact that it takes some communication skills to create quality surveys27 that yield the results you need, and not only what you want. My opinion is that smaller surveys are better than larger ones when it comes to security awareness. Most people in your organisation are not as passionate about security as you are, so making them answer a long survey is usually harder than having them answer three or four questions.
An alternative to surveys are interviews. Interviews require more resources (they take time and are one-on-one) than surveys. The upside with interviews is that you can pick up other information from the participant, and you may discover information otherwise kept from you.
Interviews can be conducted in a number of different ways, depending on your purpose. You can do the coffee-machine interview with a great number of people, where you will ask a couple of questions in an informal way to random people you meet. This may help you discover issues you are not currently aware of, and may be conducted over some time. These kinds of interviews are cheap (you conduct them when you fetch your coffee), and are best applied to collecting informal data (i.e. what you learn may not be very useful for in-depth analysis) that you can use for careful correlation with other data sources.
You may also do formal types of interviews, where you will have a defined set of questions, and where you select the participants based on what you are setting out to learn. You may interview department managers to discover discrepancies in culture between departments, or you may interview all members of a team to learn the team’s combined understanding of security.
In addition to internal data sources, you may look for information outside your organisation. Some countries and industries collect security information and create trend analysis reports that may be used to discover how your organisation compares to the industry. You may also use breach report data available for download, and compare it with your own systems and breaches.
Take the opportunity to jump into your own logs and systems. Let the question “How can I see behaviour of my users in this particular log?” guide you through your quest. You may end up becoming another great data analyst, or you may decide that you need someone else to do this job. No matter what you decide, I guarantee you that you will find ways to track behaviour. Then ask yourself the next question: “What other logs can I combine this with, and what will I then learn?”
A word of warning: you may find yourself digging deeper and deeper, and forgetting about why you are looking for a particular dataset. There is so much to be discovered in the logs!
In the next chapter I introduce the Security Culture Framework, and present one way to set up a security culture programme that yields results.
26 There are a number of methods, techniques and principles on measuring behaviour and change available to social scientists. The UK’s Government Social Research office published a report describing models for measuring behavior change in a report entitled “Reference Report: An overview of behaviour change models and their uses” (2008).
27 Surveys are a common method of conducting research, but need to be carefully composed if they are to provide meaningful results. Ideally, a statistician will be available to ensure that results can be appropriately derived from the responses, but this is largely useless unless the correct questions are being asked. Most introductory texts on research methods should have good advice on composing effective surveys.