CyCraft AIR Achieves Analytic Detection Coverage in All 20 Major Attack Steps in MITRE Engenuity’s Latest ATT&CK Evaluations

Taipei, Taiwan — 3 May 2021 — Results from the third round of the MITRE Engenuity ATT&CKⓇ Evaluations prove again that CyCraft customers are protected and secure from the most sophisticated real-world advanced persistent threats. The CyCraft AIR platform provided precise, actionable intelligence on each of the 20 steps of the evaluation with zero delayed detections and zero configuration changes to its detection logic, platform display, or endpoint sensors.

CyCraft AIR was pitted against emulations of two highly sophisticated adversaries of the financial and hospitality sector — Carbanak and FIN7. MITRE Engenuity monitored and documented CyCraft AIR’s performance against the two simulated cyber attacks through 20 major attack steps — broken further down to 174 substeps — on a Linux and Windows environment solely defended by CyCraft AIR.

CyCraft’s ATT&CK Evaluation results demonstrate how the CyCraft AIR platform uniquely secures enterprises via its dual cadence of patent-pending continuous automated forensic analyses, eliminating alert fatigue and providing precise clarity of visibility into an enterprise’s entire cyber situation.

‍

Enterprises can benefit from CyCraft AIR through its many achievements in this ATT&CK evaluation round, including:

• CyCraft AIR provided Analytic Detection (the highest detection classification) coverage through the entire attack by providing precise, actionable intelligence in each of the 20 major attack steps of the evaluation. CyCraft AIR provided enriched Analytic Coverage for each major step, not just raw telemetric data.
• CyCraft AIR delivered out-of-box detection and response with zero delayed detections, and zero configuration changes its detection logic, platform display (UX), or endpoint sensors (data sources).
• CyCraft AIR detected 100% of all attack steps targeting Linux systems while providing the highest Analytic Coverage possible, demonstrating CyCraft AIR’s effectiveness in securing cloud infrastructure.
• CyCraft AIR responded with the highest signal-to-noise ratio, 96.15%, eliminating alert fatigue through clear and precise automated intelligence reports.

‍

Out-of-Box Detections, Zero Configuration Changes & Zero Delays

Real-world environments generate massive amounts of noise — legitimate user behavior producing telemetry data; this, combined with innumerable alerts, false positives, and manual alert validation, constantly drains limited SOC and IT resources.

Two core issues to consider when mapping ATT&CK Evaluation results — which occurred in a clean environment — to the grittiness of a real-world environment are configuration changes and the quality of detections.

Here is a chart ranking all twenty-nine vendors by the total number of configuration changes and delayed detections across all 174 attack substeps. In this chart, the fewer configuration changes and delayed detections, the better.

‍

‍

CyCraft is one of the very few vendors who came prepared with a reliably and consistently fast, out-of-box solution requiring zero configuration changes.

‍

What exactly are configuration changes?

After the initial execution of the attack scenario is completed, vendors have the opportunity to alter their solution in one of three ways: detection logic, UX, and data sources. This means that 24 out of the 29 vendors manually altered the investigation flow of their solution (detection logic), altered the display of data that was already collected but in its original state did not make it visible to the user (UX), or altered the sensor to collect new information (data sources).

Vendors can opt to perform configuration changes after the initial execution of attack scenarios. Cybersecurity solution vendors could have performed a configuration change for a multitude of reasons, including their tools initially failing to detect the attack substep. While configuration changes do occur in environments over time to advance capabilities, they provide a substantial hurdle to quickly investigating and responding to an attack.

Tools that don’t detect out-of-the-box means manual configuration, which counters the idea of automating the detection and response process in an already noisy real-world environment.

From day one of its inception, CyCraft AIR was designed as an out-of-box solution ready to detect and respond in any environment, therefore requiring zero configuration changes.

It is advisable to consider vendors that did not make configuration changes during the evaluation, or when considering a vendor who did make configuration changes, learning what those were, where they occurred in the attack, how those configuration changes altered their results, and what would need to be known and at which stages of an attack to implement the UX, data source, or detection logic configuration change to gain access to a potential detection as one step in a manual investigation.

• 9% of all the detections made by all 29 vendors were due to configuration changes.
• Out of all the detections made due to configuration changes, 17.6% resulted in Telemetry detections, 9.6% resulted in General detections, 4.9% resulted in Tactic detections, and 67.9% resulted in Technique detections.
• Out of the 24 vendors who performed configuration changes, each averaged 28 configuration changes, with a range spanning from 4 configuration changes to a total of 88 configuration changes.

Only five vendors had zero configuration changes to their solution. Here is a chart ranking vendors by the total number of configuration changes made — here, lower is better.

‍

‍

CyCraft is among the few vendors that came prepared with an out-the-box solution requiring zero configuration changes to detection logic, UX, or data collection via endpoint sensor.

Delayed detections, defined as detections that were not immediately available to the analyst, also represent a crucial component in investigations: If the detection is delayed, then that upsets the investigative workflow for analysts, slowing investigation and thus response times, leaving defenders more at risk.

Twenty-one of the twenty-nine vendors had delayed detections. Here is a chart ranking vendors by the total number of delayed detections made — here, the fewer delayed detections, the better.

CyCraft is among the few vendors that came prepared with a reliably fast solution that provided zero delayed detections.

‍

Clearest Signal

The MITRE ATT&CK evaluations took place in a clean environment where vendors have a good idea of what tactics and techniques attackers will use. In a real-world environment, vendors would not know that an attack was coming and would not know what tactics and techniques would be the focus of the attack — combined with the fact that in a real, live environment, defenders would have a lot of “noise” obfuscating the signal: legitimate user behavior producing telemetry data from the endpoints.

Several vendors take a “record everything” approach, which is the same reason why SIEMs have been seen as less than ideal in the detection and response context. When taking in so much data, much of it low value to forensic investigations, the investigation process is hampered: it is slowed down, and some things just get lost in the sea of “telemetry”.

A telemetry detection means that you would have to be actively seeking out that information in order to form a detection. Telemetry detections and the term “visibility” often cause a lot of confusion, because these two things essentially amount to raw logs, which can be a lot of noise to dig through as you will have telemetry on all the legitimate events and entities as well. An attack will often comprise less than 0.001% of your endpoint telemetry data. So If a vendor claims something like “complete visibility” defenders may find themselves digging through a mire of raw logs when using that product.

To shed more light on this, we analyzed the data provided by MITRE Engenuity and present the signal-to-noise ratio, which is taken as the percentage of detections sans configuration changes that are Analytic Detections. This shows how much actionable signal you can expect from a vendor.

‍

‍

CyCraft delivers the highest fidelity signal out of all the vendors because of the UX and forensic focus of our CyCraft AIR platform.

‍

The UX of Cyberdefense

At CyCraft, we focus on the end user experience by delivering automated MDR reporting which enables defenders to completely understand an attack, visually and event-wise, and how to remediate it, without learning new terminology, querying, validating and triaging alerts, working on a workbench or the rest of what goes into typical SOC alert workflow and investigations, because we automatically perform the investigation for you with our continuous, automated, AI-driven DFIR platform.

With CyCraft AIR’s automated MDR reporting, we would have presented the attack in one report with information to remediate. Simpler than any other vendor. Zero querying. Automatic investigations.

‍

The Logic Behind the CyCraft AIR Platform

CyCraft AIR was designed and built from day one to provide enterprises with precise visibility across all the endpoints in a single environment and beyond. CyCraft AIR achieves this by uniquely analyzing continuously collected intelligence via dual cadence of patent-pending continuous automated forensic analyses, which first monitors behavior on the endpoint level and then on a full-site, cross-endpoint level. CyCraft AIR’s novel dual-cadence system allows correlated alert triage automation in the context of the entire environment and attack story line, eliminating time wasted on alert validation and triage.

The reasoning behind the dual cadence system is that often can be the case where something that appears higher priority when viewed in an isolated context, is actually lower priority when viewed in a larger context (e.g., a new, unidentified program running on a host turning out to be a game). Conversely, an alert for something appearing to be a lower priority in an isolated context should be of higher priority when placed in a greater context (e.g., malicious file transfers). CyCraft AIR’s dual cadence DFIR system automates this commonly seen investigative context switching and linking, putting effectiveness back in the hands of defenders, as opposed to other tools, which are manual, time-consuming, and error-prone.

CyCraft AIR goes beyond detection and response dashboards and workbenches by providing automated incident response reports, highlighting critical priority threats automatically triaged while also providing precise actionable step-by-step intelligence leading defenders from response to elimination, remediation, and finally resilience, reducing the risk of similar attacks.