Why “good enough” could kill your business

There are scenarios in which opting for the best possible solution is non-negotiable. Think of medical surgery, aerospace safety, military operations, or pharmaceutical development. The reason? While the probability of a catastrophe may seem low, its impact is extraordinarily high. In cybersecurity, breaches are no longer a low-probability event, and the damage from undetected or late-detected malware or phishing attacks is both severe and on the rise. So, why settle for a second-best—or worse—solution?

Effective Defense Requires 0-day Detection & Response Capabilities

Modern security is team play. Mature defense combines different technologies, approaches, and vendors to address all NSF CSF core functions: Govern, Identify, Prevent, Detect, Respond, and Recover. One important role in that mix is Advanced Threat Detection (ATD, aka “Sandboxing”), as it is the only effective solution for detecting 0-day malware and phishing threats and automatically generating Cyber Threat Intelligence (CTI) on a large scale.

Sandboxes are used by CERTs, SOCs, and CTI teams for various purposes, such as 0-day detection, alert triaging/verification/enrichment, incident response, or on-demand CTI generation. The vendor market is mature, with various players, some dedicated entirely to this problem, while others offer built-in functionality in large platforms. Whether a point product or integrated capability, most solutions rely on the same old approach and often the same open-source technology. Attackers are well aware of this, and modern threats successfully evade these solutions, resulting in low detection efficacy and low data quality.

We will describe the consequences, their impact on overall cost and risks, and why a good enough sandbox is not enough.

‘Good Enough’ Sandbox Solutions Fall Short Against Modern Threats

In a nutshell, sandboxing is an automated deep analysis of suspicious content to produce two outputs:

Verdict: Is the analyzed content malicious or not?
Analysis: Insights on the threat’s functionality and capabilities

The concept and first implementations emerged two decades ago when the number of new malware variants was exploding, necessitating new ways to detect these never-before-seen threats and automatically generate insights to cope with them in a timely manner. The situation has not changed and has actually worsened in terms of the number and complexity of new daily threats. Since sandboxing is the most effective way to address this challenge, it has become an indispensable part of modern defence, pushing attackers to evolve constantly and come up with new evasions—unfortunately, often successfully.

One may get the impression that today we are back to square one, with attackers always one step ahead and reactive defenders always one step behind. This is true only for conventional sandboxing, and the reason is obvious: the underlying technology was designed for non-evasive Windows executables in clearly defined, non-customizable detonation environments. Since the mid-2000s, malware has dramatically evolved. Today, modern malware is very different in multiple ways. It

Comes in all conceivable formats and complexities,
Combines different payloads in consecutive stages,
Requires specific user interaction to reveal full functionality,
Is organized in multiple protection layers,
Only operates in specific target environments,
Behaves differently depending on the victim,
Is equipped with sophisticated defense capabilities,

Even with such modern malware, each sandbox will still produce some verdict and analysis. Unfortunately, often the verdict is wrong, and the analysis is incomplete and noisy, i.e., critical information is missing, and signals are hidden by the massive amount of irrelevant data.

Unlike other software domains where you can assess tool accuracy by comparing its output with the ground truth, there is no such truth for unknown threats. Dealing with wrong and incomplete information seems like an efficiency, ROI, or cost problem. However, the reality is worse, as it dramatically impacts the risk and survivability of the affected organization.

Ineffective 0-Day Threat Detection

Two things are clear:

Modern defense requires effective detection capabilities to complement preventive measures.
Time to Detect and cost of a breach are directly related.

One obvious consequence is that sandbox detection efficacy is key, i.e., producing correct verdicts. There is a related pitfall when comparing the efficacy of sandbox solutions. Sandboxes often utilize signatures of known threats to accelerate and enrich their processing. This is best practice and provides additional value. However, this only works after new threats become known and does not provide detection or insights when they are first used. Thus, when comparing sandbox solutions in a Proof of Concept (POC), it’s essential to test unknown malware and verify that detections are based on deep analysis output, not on signatures. Otherwise, the solution is just a slow and more expensive antivirus.

The verdict is crucial, but higher maturity security teams also use sandboxing for Detection Engineeringand Threat Hunting. Both need high-quality Indicators of Compromise (IOCs) to build and test rules. Quality is indispensable: without the right IOCs, you won’t catch anything; with the wrong IOCs, your rules will produce many false alerts. For 0-day detection, it’s critical that the analysis contains all relevant IOCs and can differentiate them from the noise.

Slow Response Failing to Mitigate Incidents

After suspicious activity is alerted and confirmed, the incident response starts, and the Time to Respond becomes crucial. The longer it takes to mitigate a threat, the more impact and damage it causes, such as exfiltrating, encrypting, or destroying sensitive data, or infecting other machines through lateral movement. The response process is hindered by incomplete analysis and missing indicators, and it can be significantly slowed down by noise, which must first be removed. This identification and separation of noise from signal is a time-consuming process impacted by the clarity of the report and the experience and skill level of the person doing it.

In addition to the time needed for cleaning up the data, response teams are often misled by chasing irrelevant or incorrect artifacts instead of immediately focusing on the real and relevant IOCs. A related problem is the incompleteness of many sandbox-generated reports.

As mentioned, modern malware comprises multiple stages and layers that often unfold only after a certain time, triggers, or user activity. When an analysis only produces insights on the first stage or layer, the resulting analysis will lack fundamental data points needed to respond appropriately. Only after valuable time has elapsed do security teams realize they are missing important information and then need to go back to the analysis part before they can finally mitigate or stop the threat.

Wrong Data Leading to Fatal Decisions

A related risk is being misled on the severity of a particular incident. If your analysis covers only the first stage or layer and lacks the subsequent ones, it does not provide the full picture. Security teams may identify an already known and easy-to-block first-stage downloader or dropper but remain unaware of later stages that may contain a targeted, dangerous, or novel threat. This incomplete view misleads on the severity of the incident, provides a false sense of security, and leads to impactful wrong decisions.

Operationally, you may think, “Hey, I am not affected by this,” if you only search for incomplete or incorrect indicators. Or you may think, “OK, I am affected, but it is not that bad.” Worse is the impact of incomplete and wrong data on strategic decisions, leading to a misjudgment of your capabilities and current security posture. This may result in the wrong focus and strategic view on where to progress your security program and maturity.

Low-Quality Data Inhibits Automation and Team Efficiency

All security teams today face a skill shortage and need to optimize their efficiency. Utilizing sandbox solutions in CERTs and SOCs sounds promising, as it automates manual security tasks, accelerates work, and enhances the capabilities of less-experienced team members. Additionally, it equips everyone with Threat Intelligence to guide decision-making.

The reality, however, often differs: low-quality data is unsuitable for automation or decision-making. It first needs to be validated and cleaned, costing additional time, disrupting automation, and requiring comprehensive experience to handle the lack of clarity. Especially when using AI/ML-based automation systems, data quality is critical. Since their logic and decision-making are typically not transparent, feeding them untrusted input data will lead to untrusted output data. Manual cleanup is not only an efficiency and cost problem. Repetitive and boring tasks burn out SOC team members, leading to high attrition and Alert Fatigue, where critical alerts are ignored or overlooked.

Dealing with noise is a challenge when working with sandbox analyses. Hence, a good report does not produce any noise. Beyond that, a great report can differentiate between correct but less distinctive artifacts and true IOCs. This dramatically reduces the turnaround time between generating and using sandbox-generated Threat Intelligence. Simultaneously, it guides security analysts and lowers the expertise required to make reasonable use of the data.

Monoculture Lowers Efficacy and Creates Dependency

The ongoing trend of vendor consolidation promises cost-cutting and better convenience by providing all data and controls through a single pane of glass. While this sounds compelling, it also dramatically increases risk due to lower efficacy and dependency.

Every security vendor has strengths, focus areas, weaknesses, and blind spots. This includes technological, organizational, and political areas. Strong and complete defense combines different technologies, perspectives, and approaches to increase coverage and efficacy. Diversity ensures that all strengths and focus areas are covered: effective security is a team effort. Additionally, security teams often need or want a second opinion for decision-making and prioritization. Relying on a second opinion based on the same technology or approach is largely useless. Combining different perspectives allows for faster and more trustworthy alert validation and decision-making.

Another important aspect is self-defense capabilities. Consolidating all security products into one vendor makes you entirely dependent on that vendor. In a crisis, you will have limited visibility or control over their strategy, tech stack, prioritization, ability, or willingness to help you promptly. Thus, it is critical to build in-house capabilities and expertise for mission-critical use cases. This includes preventing vendor lock-in and avoiding complete dependence on a single third party.

Always One Step Behind

Attackers never sleep and continuously find new ways to compromise systems and remain undetected. This is facilitated by an ever-increasing attack surface—both in size and complexity. There are billions of devices connected to the Internet, using different hardware, operating systems, software stacks, and patch levels. Even in standard environments (Windows, Mac, Linux, Android), attackers constantly develop new attack vectors and evasion techniques. More recent examples include:

To cope with this creativity and speed, security vendors must constantly track and anticipate changes, integrating new capabilities and technologies into their solutions. This necessitates 100% focus and relentless innovation, which is particularly challenging for platform vendors that combine many different capabilities into one platform.