The Fault in Our Metrics. Rethinking How We Measure Detection & Response
Your metrics are boring and dangerous. Recycled slides with meaningless counts of alerts, incidents, true and false positives… SNOOZE. Even worse, it’s motivating your team to distort the truth and subvert progress. This talk is your wake-up call to rethink your detection and response metrics.
Metrics tell a story. But before we can describe the effectiveness of our capabilities, our audience first needs to grasp what modern detection and response is and its value. So, how do we tell that story, especially to leadership with a limited amount of time?
Measurements help us get results. But if you’re advocating for faster response times, you might be encouraging your team to make hasty decisions that lead to increased risk. So, how do we find a set of measurements, both qualitative and quantitative, that incentivises progress and serves as a north star to modern detection and response?
Metrics help shape decisions. But legacy methods of evaluating and reporting are preventing you from getting the support and funding you need to succeed. At the end of this talk, you’ll walk away with a practical framework for developing your own metrics, a new maturity model for measuring detection and response capabilities, data gathering techniques that tell a convincing story using micro-purple testing, and lots of visual examples of metrics that won’t put your audience to sleep.
AI Summary
Key facts
- Metrics often start from a flawed point, with previous managers fabricating data, leading to a cycle of poor metric-driven decisions.
- The security field has matured significantly, yet many organizations are stuck using outdated metrics that do not accurately reflect their operational effectiveness.
- Allyn Stott advocates for a shift from traditional metrics to more insightful, actionable metrics that truly reflect the security posture and operational efficiency of an organization.
Ideas
- The importance of selecting the right metrics in security operations to avoid basing decisions on incorrect data, which can lead to a cycle of poor decision-making.
- Introduction of the SABER framework (Streamlining operations, Awareness raising, Vigilance measuring, Explorations through networks, Readiness in incident response) to guide the creation of meaningful security metrics.
- The concept of the Threat Detection and Response (TDR) maturity model, which helps organizations assess and plan the development of their detection and response capabilities across different areas.
Keywords
- Security Metrics
- SABER Framework
- TDR Maturity Model
- Detection and Response
- Metric Improvement
Quotes
- “Metrics Drive improvements...what if you're measuring the wrong thing?”
- “Metrics are an annoying PowerPoint I need to update every month.”
- “When you look at a metric it should say what do you want from me.”
- “You become what you measure.”
Recommendations
- Organizations should evaluate and possibly overhaul their current security metrics in favor of metrics that provide actionable insights and truly reflect their security posture.
- Adopt the SABER framework and TDR maturity model to guide the development of security operations and metrics, ensuring they are aligned with organizational goals and the current threat landscape.
Resources
- SABER Framework — Framework
- Threat Detection and Response (TDR) Maturity Model — Model
About the speaker
Allyn Stott
Allyn Stott is a senior staff engineer at Airbnb where he works on the infosec technology leadership team. He spends most of his time working on enterprise security, threat detection, and incident response. Over the past decade, he has built and run detection and response programs at companies including Delta Dental of California, MZ, and Palantir. Red team tears are his testimonials.
In the late evenings, after his toddler ceases all antics for the day, Allyn writes a semi-regular, exclusive security newsletter. This morning espresso shot can be served directly to your inbox by subscribing at meoward.co.
Read more …