How to Define and Build an Effective Cyber Threat Intelligence Capability (2015)
Chapter 7. Who: Given Why, What, and How, Now You Can Ask Where To Get It
At this point, the book has covered why you are building a cyber-threat capability, what you are going to need, and how you plan to implement it. The next chapter covers going out to the market and deciding who to buy from and the six key axes on which to evaluate an intelligence or data vendor.
chief financial officer (CFO)
By this time in the process, you know – technologically and in terms of personnel – why you are doing it, what you are going to need, and how you plan to implement it. The next logical step from here is going out to the market and deciding whom to buy from. Now there are a lot of ways to compare vendors, but you should follow the pattern of looking first at the offering and then at the vendor themselves. For an offering, service, a source of data, a feed or intelligence, going for a pilot, or a trial will be helpful, but only if the client and vendor can agree on a framework for feedback and some concrete measures or axes of evaluation. If you intend to request service at no charge, a fair trade between buyer and supplier is the exchange of feedback for a brief period of service, and using an agreed framework for what defines “success.”
To effectively make the best choice, there are at least six axes on which customers or prospects should evaluate an intelligence or data vendor.
1. Quality: This is a tricky aspect because how it is measured varies from feed or data type to data type. Therefore, there must be some way to measure quality and you should agree on it before the trial.
2. Quantity*: This one comes with an asterisk. Sometimes quantity is an indicator of value, since in many cases either more is better or less is better depending on the data in question. It is also quite possible that quantity is not actually applicable, since for example, you might have a feed that is extremely low volume, but even the occasional finding is of incredibly high value. For this element, you have to always look at the data type to understand whether volume is a valuable axis of comparison.
3. Uniqueness: Imagine if one vendor has good data but so do three other. If their offerings overlap by 85%, your decision has to be based on other factors as well. For example, you may get 85% of the same data from two vendors, but one guy may offer it 36 hours ahead of the other guy. If all other things are equal, in security, sooner is usually better, but if the data sets are different, uniqueness can be a key discriminator.
4. Value: When it comes to value, you may equate it to cost or it may have another axis of its own. If all vendors’ offerings are functionally equal in every material respect, then lower price should win.
5. Ease: If one vendor’s offering comes in a standard compliant JSON format that can be ingested with two scripts whereas another set is written in a proprietary flavor of XML which requires the efforts of two developers for a week, which one would you choose? Remember that effort costs time and money, so ease of implementation is a vital aspect of “value” to consider along with price.
6. The vendors themselves: So far we have mostly spoken about the product. You also have to look at the vendor. To be very candid, there are a lot of great things going on out there; a lot of neat technologies and many really cool companies doing really interesting things. However, if you are going to construct mission critical activities or risk mitigation plans around some of those vendors, you need to be sure they are going to stick around. The great thing about small and new companies is they do great things. The bad thing is that four out of five of them fail very quickly. You need to have a look at the vendor’s reputation, their customer base, their longevity, or their age in the industry.
Invest your time in discovering the answers to important questions such as:
• Are they stable?
• Are they profitable? Running out of cash?
• What source of funding got them this far? Customers? Bootstrapping by the owners? Venture capital?
• Do they themselves have the kind of security you would expect from a security vendor?
• Are they audited?
• Do they have policies and procedures?
• Do they pen-test themselves?
• Have they ever been breached?
• If you need expertise skills or knowledge, how deep is their bench?
7.1. Reporting and management communication
Finally, before concluding this book, a quick note about reporting and management communication is definitely needed. You have to communicate to management the value, the operational importance and the mission of what you are doing.
A lot of folks fall down when they start delivering reports around nominal metrics. In this case, they provide numbers around simple volume metrics, for example, the number of searches run, or the number of alerts that were triggered, or items investigated, or the number of documents that were found, tagged, discovered, or quarantined. The problem is that none of those translate directly to a mission outcome. Even if they do, you have to spell that out for management.
If those are nominal metrics, what are meaningful reports? Those would usually contain important data, such as:
• Time to discover an issue
• Time to remove the causes of a problem and remedy it
• The severity, not the number of incidents identified
• The number of root causes (which is important so that the nominal items do not keep growing or are reported)
Finally, meaningful metrics then have to translate to mission outcomes that can also be reported. For instance, if egress of sensitive data is uncovered, what actions ensue that can then be reported as concrete changes undertaken to reduce impact and future risk? If you identified an insider threat, did you change classification rules for data loss prevention (DLP) and egress monitoring? Have you instituted a policy change that should be communicated to the organization as a whole? Has HR been engaged or any personnel action taken? These are business outcomes that translate back to management of the business; nominal metrics alone will not make the leap to management communication effectively.
7.2. Defining and articulating budget needs
Finally, if you know why you are doing it, what operational activities you will support, what intelligence needs you have, how you are going to implement it, and which vendors you want to buy it from (or at least what rough costs are for the tools and services you will need to buy), you are now ready to put together either an up-front or ongoing budget request.
If you are in the pre-operational stage, you will need to speak to the CFO to ask for both capex and opex. That is money to build the system and money to run it. To a CFO, those are two very different things. If these words or concepts are foreign to you, make sure to ask someone in finance or management to help you. If you only speak tech, and they only speak accounting, your budget request may face a rough road ahead.
Linking this concept back to the previous section, the managerial reporting you define will help dictate the program’s long-term survival. Every budget cycle is bound to bring questions about the money spent and whether it is worth spending.
You have to be able to say, “Yes, we should still keep doing this and here is why.” You should also be able to translate that back to the business. You need to make sure you have the right people to build and run a threat intelligence capability, and make clear what those people cost.
Another timely concern at this moment – you should know that given the current state of the market, those capabilities (in both building and operating a threat intelligence capability) are in great demand. To ensure that your operations do not get affected, consider looking at outsourcing, contractors, or training to improve the abilities of your own people, making an investment in them to help ensure that they stick around. For training, always choose the most loyal people, since the last thing you need halfway through the project is for someone to offer him or her a little more money, leaving you with a half-finished project and a newly created expert who just left for a competitor.
Last, but by no mean least, factor risk into your vendor evaluation. This is not only on the data side, but also on the provider. You have to evaluate the company, not just the offering, when ensuring the best and safest expenditure of your limited budget.