Methodology

detection_rate = (detected + worker_killed) / (detected + worker_killed + blocked + not_detected)

Each trigger test ends in exactly one outcome. What each one means, and which side of the fraction it lands on:

Excluding errors and skips from both sides matters: counting them as misses would punish a machine for a test that never ran, and counting them as detections would flatter it. A run in posture_only mode triggers nothing at all, so it contributes no tests and cannot move any detection rate.

Where a denominator is zero, the figure is shown as an em dash — never as 0%. Zero percent is a finding; no data is not.

Miss rate is not_detected over the same denominator. Miss rate and detection rate do not have to sum to 100%, because a test can also be blocked, which is neither.

A group is only published once 3 or more distinct public reports back it. Below that the group is withheld and the page says so, rather than quietly showing a shorter list.

This is a privacy control, not a statistical nicety. With one or two contributors an “average” is just one company's posture with a percent sign after it, and anyone who knows what they run could read their result straight off the page.

The threshold applies to every grouped figure independently:

• every row of the technique, product, platform, OS, posture and policy tables;
• every month on the trend chart;
• every individual cell of the product × technique heatmap — a single cell identifies as readily as a single row, so a thin cell reads “no data” even when its row and column are both published;
• the filter dropdowns themselves — a vendor below the threshold is not offered as a filter option, because the list of choices would otherwise disclose that somebody runs it;
• every CSV and JSON export, which is built from the same aggregates as the page.

The threshold is the public_stats_min_reports setting, currently 3. Raising it withholds more; it can never publish less than the pages already show.

Deleted reports are excluded from the moment they are deleted, including from cached aggregates. Changing a report to private removes it from every figure here.

No aggregate on this site selects, stores or displays an identifying field. Hostnames and serial numbers arrive already hashed by the client and are never shown; IP addresses are hashed on receipt; organisation names, account names and individual report identifiers do not appear in any statistic. Your own reports are the one exception, on your own comparison page, which is visible only to you.

A report records which protection products were installed, and what the machine did when attacked. It cannot attribute a specific detection to a specific vendor — no report says “the EDR caught this, the AV missed it”.

So every product row means: on machines where this product was installed, this share of simulated attacks was detected by something. A machine running an EDR and an AV contributes its results to both rows. Where three products are installed, all three get the credit and all three get the blame.

Configuration, licence tier, policy and how long the agent had been deployed all matter at least as much as which product it is, and this sample cannot separate them. A low row is a prompt to ask why, not proof that a product is bad.

Where the client could not attribute a detection to a single vendor — several vendors ship the same OEM anti-malware engine driver — it reports the candidates in the product's notes rather than guessing, and that note travels with the product wherever it is displayed.

The posture score is a 0–100 weighted roll-up of what the machine actually reported: EDR and AV present and healthy, Defender state, firewall, disk encryption, patch currency, local-administrator hygiene and policy compliance. It is deliberately independent of the detection tests — posture is what is configured, detection is what happened.

The client computes a score and the server recomputes its own from the same raw report, so a change to the scoring model applies to every historical report at once. Where the two differ, every figure on these pages uses the server's.

Grade bands: Grade A ≥ 90 · Grade B ≥ 75 · Grade C ≥ 60 · Grade D ≥ 40 · Grade F < 40.

Alongside the trigger tests, a run evaluates a set of CIS-aligned hardening controls — audit policy, script-block logging, SMBv1, LSA protection, SSH hardening, sudo rules, secure boot, and so on. Each ends in one of four results, and only two of them are counted:

pass_rate = pass / (pass + fail)

• pass / fail — the check was read and compared against the benchmark. Both sides of the fraction.
• unknown — the check could not be evaluated: no administrator rights, a missing command, output that could not be parsed. Excluded from the rate entirely.
• not_applicable — wrong platform or role for the check. Also excluded.

Counting unknown as a failure is the obvious mistake here, and it would be a serious one: it would make every unprivileged run look like a badly configured host, and it would reward a machine for being unreadable. Reports say how many checks they could not read, and that number is published next to the rate rather than folded into it.

The framework view maps those same checks onto CIS Controls v8, ISO/IEC 27001:2022, NIS2 Article 21 and SOC 2 control identifiers. A control counts as failed when any check mapped to it failed — partial compliance with a control is not compliance with it.

This is not an audit position and cannot be used as one. It is a themed mapping intended to help you find the right internal conversation. “Controls touched” counts only the controls these checks actually reach, which is a small fraction of any framework, so a high pass rate here says nothing whatsoever about the rest of it.

Every test the client runs declares the ATT&CK techniques it exercises, and the test registry refuses to load a test that declares none — so nothing can quietly drop out of the coverage view by being forgotten.

Missed techniques counts this project's tests. ATT&CK coverage counts MITRE's techniques. They are not the same numbers regrouped: several tests can map to one technique, and a technique counts as covered when at least one test exercising it was detected or blocked. A technique with a high rate can therefore still hide one test that always slips through, which is precisely why both views exist.

At tactic level, a tactic's sample size is the largest of its techniques' report counts. Every technique's contributing reports are a subset of the tactic's, so that is a lower bound — the safe direction for a suppression decision.

Technique names and tactic assignments are reference data held by this site, not by the reports: the client sends IDs only, so a corrected label applies retroactively to every report ever uploaded. Where a technique belongs to more than one tactic it is filed under the one the test is measuring.

Where the client can read the host's own logs — journald, the Windows event log, the macOS unified log — it looks for entries that match what it just did, and records the gap between triggering the behaviour and the first matching entry.

Correlation is by time window and indicator. That makes a match evidence, not proof: a log line that arrived just after a trigger and mentions the right file or process is very probably about it, but nothing here can prove causation. The clock is the defender's, not ours.

• A matched entry can upgrade an outcome from not_detected to detected. Something did notice, even if nothing visibly reacted.
• A test with no matching entry is absent from the latency figures rather than counted as slow. Unknown is not the same as bad.
• Where the client could not read the logs at all, the whole section is missing from the report and it contributes nothing — which is different from a host that logged nothing.

Response times from a SOC are a separate matter: those are supplied by hand after a run, and an unanswered question stays unanswered rather than becoming a “no”. They are self-reported and unverifiable by the client, which is why they are never mixed into anything the machine measured.

Everywhere else on this site a detection is the good outcome. On egress the good outcome is blocked: the run tries to reach internet-wide scan databases, exploit archives, malware repositories, offensive-tooling mirrors, breach-data services, anonymisers and consumer cloud storage, and a destination that stays reachable is the finding.

Each category is attempted twice — once as a DNS resolution and once as an HTTPS request — because those are two different controls. A network that filters only at its resolver is bypassed by any host using its own DNS, and the report records which of the two stopped the attempt so the two cases can be told apart.

Blocked and detected are both counted as “the network handled it”: something either refused the name or refused the request. The complement, reachable, is what the category rate reports.

The destination catalogue is public, well-known security and research services. Criminal marketplaces and onion services are deliberately excluded, and a test asserts they stay excluded — testing whether a filter blocks a crime scene is not worth visiting one. --list-targets prints every destination the suite can ever reach.

A report inventories installed software and the remote-access paths into the machine. Where an item is flagged, the flag says what the software is — a remote-support tool, an administrative tool, a version past its vendor support date — and never that it is vulnerable.

No CVE matching happens anywhere in this product, client or server, and none should be inferred from these pages. “AnyDesk is installed on 40% of hosts” is a statement about attack surface, not about a flaw in AnyDesk.

End-of-life claims are deliberately narrow and version-specific, because a wrong end-of-life claim in a security report is worse than an absent one: a rule only applies when the version it inspects belongs to the product itself, so a library that merely carries a runtime's name in its package name is not flagged on the runtime's behalf.

Filter state lives in the address bar, so any view can be shared or bookmarked and it will show the same slice to the next reader: platform, test category, protection vendor and product, grade, period and sort order. A filter can only ever narrow the public sample — there is no parameter, on any page, that widens it.

Periods available: all time, last 30 days, last 90 days, last 6 months, last 12 months, last 24 months.

Every aggregate table can be exported, with the same suppression applied:

• Technique difficulty — CSV · JSON
• Protection product effectiveness — CSV · JSON
• Detection by test category — CSV · JSON
• Detection by platform — CSV · JSON
• Detection by operating system — CSV · JSON
• Posture grade distribution — CSV · JSON
• Posture features — CSV · JSON
• Policy compliance — CSV · JSON
• Detection over time — CSV · JSON
• Product × technique matrix — CSV · JSON
• ATT&CK technique coverage — CSV · JSON
• ATT&CK coverage by tactic — CSV · JSON
• Hardening control checks — CSV · JSON
• Compliance framework controls — CSV · JSON
• Web-filter categories — CSV · JSON
• Trigger-to-log latency by test — CSV · JSON
• Software found most often — CSV · JSON
• Remote-access exposure — CSV · JSON

The JSON export carries the active filters, the suppression threshold, the number of withheld groups and the bias notice alongside the rows, so a figure cannot be lifted out of context by accident.

If a figure here looks wrong, it may well be. The aggregates are recomputed from the stored reports every time, so a bug is fixable and a fix applies retroactively to everything.

Report schema and outcome definitions: docs/REPORT_SCHEMA.md in the project. The client is open about exactly what it triggers — see how to run it.