There are two general claims I would like to make about the state of cybersecurity that I believe to be true and connected. The first is that the field is rife with pseudoscience and lacks an appreciation for empirical methods. The second is that much of the bad information in the field has originated from industry players that have incentives for certain beliefs to take hold. I suppose I am in the right field as exposing exploitative bad science has been a passion of mine.
IBM’s recent Cost of a Data Breach Report presents some interesting information, but also makes dubious claims that are worth examining. Based on the report, data was collected by the Ponemon Institute using a confidential survey tool/approach to gather information from 553 organizations that experienced a recent breach. There are some major methodological limitations (some of which are listed in the report) that ought to prevent unreasonably strong conclusions. Nevertheless, IBM’s press release for the report leads with the following subtitle:
The full report provides additional AI/Automation conclusions, claiming that “Extensive security AI and automation use delivered cost savings of nearly USD 1.8 million.” This is presented as the average cost saved per breach. Confidently attributing these savings to a class of technologies would require an in-depth analysis and understanding of all the mechanisms involved or an experimental design that could account for the numerous variables (confounders) at play. Neither of these occurred.
Based on the limited reporting of methodology, these conclusions draw from estimated breach costs correlated with self-reported (by the surveyed organizations) level of AI/Automation use, using only the categories “No use,” “limited use,” and “extensive use.” At no point was “AI/Automation” well-defined nor was there any examination of how these technologies were employed or what role they played in detecting and mitigating breaches.
Such a fundamental error – mistaking correlation for causation – is unacceptable as science at the high school level. Yet, not only is this practice normalized by major industry players in cybersecurity, but the surrounding media ecosystem is entirely uncritical (just search for related reporting). A more responsible conclusion would look something like this:
Organizations that reported greater use of any form of AI/Automation experienced less impactful breaches on average; however, it cannot be concluded whether the use of AI/Automation itself was a causal factor. Therefore, further research is necessary to investigate the use of AI/Automation during breach lifecycles.
Without a deeper analysis, this is really all that can be concluded. There exists many possible explanations for the observed data that do not require attributing any benefit to AI/Automation technologies. For example, it is quite likely that organizations with more mature security teams and more funding are making more extensive use of these technologies. It could very well be other practices developed and employed by these mature teams that are primarily responsible for the reduction in breach impact observed.
There is no indication that the methodology ruled out any (let alone all) other plausible explanations and factors, but instead of providing a grounded conclusion, the report takes a firm stance on AI/Automation as the causal factor. Why? Of the 4 recommendations concluding the report, the 3rd recommendation leads: “Use security AI and automation to increase speed and accuracy.” This recommendation (page 65 of the report) links to 2 IBM products, one of which prominently markets the use of “AI and automation.” I cannot make the claim that IBM selling AI/Automation products is a causal factor in the conclusions they have drawn (I am not so sloppy), but it sure is suspect.
If the security industry is to reach a high level of maturity and respectability, it needs to begin taking empirical methods seriously. This includes industry, practitioners, and the surrounding media ecosystem as well. Unfortunately, it is easy to get caught up in the latest hype bubble and much harder to conduct rigorous analysis and hypothesis-driven research. Especially when it comes to “AI,” our collective low standard for evidence is already being used against workers. Don’t get me wrong; I am supportive of automation, but you will need more than hype to sell me on a solution. Show me the evidence.