At the Black Hat Europe conference in December, I sat down with one of our principal security analysts, Paul Stringfellow. In this first part of our conversation, we discuss the complexities of navigating cybersecurity tools and defining relevant metrics to measure ROI and risk.
John: Paul, how does the end user organization make sense of everything that’s going on? Here at Black Hat, there are a lot of different technologies, options, topics and categories. In our research, there are 30-50 different security topics: position management, service management, asset management, SIEM, SOAR, EDR, XDR, and so on. However, from the perspective of the end-user organization, they don’t want to think about 40-50 different things. They want to think about 10, 5 or maybe even 3. Your job is to deploy these technologies. How do they want to think about it, and how do you help them translate the complexity we see here into the simplicity they’re looking for?
Paul: I look forward to events like this because the challenge is so complex and rapidly evolving. I don’t think you can be a modern CIO or security leader without spending time with your suppliers and the wider industry. Not necessarily in Black Hat Europe, but you have to connect with your vendors to do your job.
Going back to your point about 40 or 50 sellers, you’re right. The average number of cybersecurity tools in an organization is between 40 and 60, depending on which research you refer to. So, how do you keep up with it? When I come to events like this, I like to do two things – and I’ve added a third since I started working with GigaOm. One of them is meeting with vendors because people have asked me to. Second, go to some presentations. The third is to walk around the show floor and talk to vendors, especially ones I’ve never met, to see what they do.
I was sitting in a session yesterday and the title caught my eye: “How to Identify Cybersecurity Metrics That Will Bring You Value.” This caught my attention from an analyst perspective, because part of what we do at GigaOm is create metrics to measure the effectiveness of solutions on a given topic. But if you’re deploying technology as part of SecOps or IT operations, you’re collecting a lot of metrics to make a decision. One of the things they talked about in the session was the issue of creating so many metrics because we have so many tools that there’s so much noise. How do you begin to discover value?
The long answer to your question is that they suggested something that I thought was a really smart approach: step back and think as an organization about which metrics matter. What do you need to know as an entrepreneur? By doing this, you can reduce the noise and also potentially reduce the number of tools you use to provide these metrics. If you decide that a certain metric is no longer valuable, why keep the tool that provides it? If it does nothing but give you this metric, remove it. I thought that was a really interesting approach. It’s almost like, “We’ve done all these things. Now let’s think about what else really matters.”
This is an evolving space and how we deal with it must evolve as well. You can’t just assume that because you bought something five years ago, it still has value. You probably have three other tools now that do the same thing. The way we approach threat has changed, and the way we approach security has changed. We need to go back to some of these tools and ask, “Do we really still need this?”
John: This is how we measure our success and change in return.
Paul: Yes, and I think that is extremely important. I was talking to someone recently about the importance of automation. If we are going to invest in automation, are we better off now than we were 12 months ago after it was introduced? We’ve spent money on automation tools, and none of them are free. We were sold on the idea that these tools would solve our problems. One thing I do in my role as CTO, outside of my work with GigaOm, is take the dreams and visions of vendors and turn them into reality for what customers want.
Vendors have aspirations that their products will change the world for you, but the reality is what the customer needs at the other end. It’s that kind of consolidation and understanding—being able to measure what happened before we implemented something and what happened after. Can we show improvement and was this investment really worth it?
John: Finally, here’s my hypothesis: Risk is the only metric that matters. You can divide it into reputational risk, business risk or technical risk. For example, will you lose data? Are you going to compromise data and thereby damage your business? Or will you expose the data and upset your customers, which could hit you like a ton of bricks? But then there’s the other side – are you spending way more money than you need to mitigate the risks?
So you get to cost, efficiency and so on, but do organizations think about it like that? Because that’s my old school view. Maybe it’s moved.
Paul: I think you are on the right track. As an industry, we live in a small echo chamber. So when I say “industry” I mean the little that I see, which is just a small part of the whole industry. But in this part, I think we’re seeing a shift. There is much more talk about risk in conversations with customers. They are beginning to understand the balance between spending and risk and trying to figure out how much risk they are comfortable with. You will never eliminate all risks. No matter how many security tools you implement, there’s always the risk that someone will do something stupid that leaves the business vulnerable. And that’s before we even get into AI agents trying to befriend other AI agents to do malicious things – that’s a whole other conversation.
John: Like social engineering?
Paul: Yes, very much. That’s a whole different show. But understanding risk is becoming more common. The people I talk to are starting to realize that it’s about risk management. You cannot eliminate all security risks and you cannot deal with every incident. You need to focus on identifying the real risks to your business. For example, one criticism of the CVE score is that people look at a CVE with a score of 9.8 and assume it’s a huge risk, but there’s no context around it. They do not take into account whether the CVE has been seen in the wild. If not, what is the risk of being the first to encounter it? And if the exploit is so complicated that it hasn’t been seen in the wild, how realistic is it that someone will use it?
It is so complicated to exploit that no one will ever exploit it. It’s 9.8 and it shows up on your vulnerability scanner and says “You really need to deal with this.” The reality is that you’ve already seen a shift where no context applies—if we’ve seen it in the wild.
John: Risk equals probability multiplied by impact. So you’re talking about the probability and then, is it going to impact your business? Does it affect the system used for maintenance once every six months, or is it your customer-facing site? But I’m curious because in the 1990s when we were doing it practically, we went through a wave of risk aversion, then we said, “We have to stop everything,” which you’re talking about, to risk mitigation and risk prioritization and so on.
But with the advancement of the Cloud and the rise of new cultures like agile in the digital world, we feel like we’ve gone back to, “Well, you’ve got to prevent that, lock all the doors and implement zero trust.” And now we’re seeing a wave of, “Maybe we should think a little smarter about this.”
Paul: That’s a really good point, and it’s actually an interesting parallel you make. Let’s have a little fight while we record it. Do you mind if I argue with you? I will challenge your definition of zero trust for a moment. So zero trust is often seen as something that tries to stop everything. That probably doesn’t apply to zero trust. Zero trust is more of an attitude, and technology can help support that attitude. Anyway, this is a personal debate with me. Target, zero confidence…
Now I’ll just cut back here later and argue with myself. So zero trust… Taking that as an example is a good one. What we were doing was implicit trust – you’d log in and I’d accept your username and password, and everything you did after that, inside the secure bubble, would be considered valid without malicious activity. The problem is, when your account is compromised, logging in might be the only safe thing you do. Once logged in, everything your compromised account tries to do is malicious. If we do implicit trust, we are not very smart.
John: So the opposite would be to block access completely?
Paul: That’s not reality. We can’t just stop people from signing up. Zero trust allows us to opt in but not blindly trust everything. For now, we trust you and evaluate your actions on an ongoing basis. If you do something that makes us no longer trust you, we act accordingly. It’s about constantly evaluating whether your activities are appropriate or potentially harmful, and then acting accordingly.
John: This will be a very disappointing argument because I agree with everything you say. You’ve argued with yourself more than I’ll be able to, but I think as you said, the castle defense model – once you’re in, you’re in.
I’m mixing two things up there, but the idea is that once you’re inside the castle, you can do whatever you want. That has changed.
So, what to do about it? Read Part 2 for information on how to ensure a cost-effective response.
The post Making Sense of Cybersecurity – Part 1: Seeing Through Complexity appeared first on Gigaom.