Skip to main content

Detection Quality Benchmark Methodology

Live results: the latest approved numbers (precision, recall, time-to-detect, per family) are published at dralvia.tech detection quality. This page explains how those numbers are produced.

Why we publish this​

A detection company that does not publish its accuracy is asking you to take its word for it. We would rather show the numbers and the method behind them. This page documents exactly how Dralvia measures detection quality, so the results are reproducible and honest rather than asserted.

Principles​

  1. Every number is regenerable. The results come from a benchmark harness anyone on our team can re-run. No hand-typed stats.
  2. Fresh, live ground truth only. We measure on URLs first seen in the last N hours that are still serving a real page at scan time. Naive benchmarks are inflated because a large share of feed URLs are already dead by scan time, and many of those still answer with an HTTP 200 to a parked, suspended, default server, or blank page. Counting those as missed detections would understate real quality, so we use a content-aware liveness check (not just a status code): a URL only counts if the page still serves real, non-parked content. The check uses generic deadness signals only, never our own detector, so it cannot bias the recall number.
  3. Methodology is published with the numbers. Honesty is the brand.

What we measure​

  • Headline precision and recall on the threat type this scanner is built for: phishing and malicious websites. The public number is gated on this family so it reflects the product's actual job. We also publish a per-family breakdown for transparency, but families the website scanner is not positioned for (for example, hosts that only serve a malware payload file) are reported as their own line and never blended into the headline, because mixing them would understate phishing detection quality.
  • Time-to-detect: the delta between when a URL first appears in a public feed and when Dralvia reaches a verdict. This is reported as a median and a p90.
  • False-positive rate on a benign control set of well-known, known-good sites, extended with rotating long-tail benign URLs.
  • Median scan latency for the default quick scan track. When a run also includes a deeper scan track, its latency and detection quality are reported separately so quick and deeper scans are never blended into one misleading number.

Ground truth​

  • Positives: fresh, labeled malicious URLs from public threat feeds, selected within a recent first-seen window and liveness-checked at scan time. Positives are balanced by threat type so the phishing family the scanner is built for is always well represented and is never crowded out by a larger, unrelated feed.
  • Controls: a benign control corpus of known-good sites plus rotating long-tail benign URLs, so a signal that also fires on legitimate traffic is caught as a false positive.

How a run works​

  1. Load fresh positives and benign controls.
  2. Content-aware liveness-check every URL; drop anything not serving a real page at scan time (dead host, HTTP error, parked, suspended, default server, or blank page). Each dropped URL is recorded with the reason, so the run shows exactly how much dead ground truth was excluded.
  3. Scan each live URL, using the full URL including its path (most malicious pages live at a path, not the bare domain), through the same scan path customers use.
  4. When multiple scan tracks are enabled, run the quick track and deeper track against the same live sample set. The approved public headline uses the configured default track, and each track keeps its own precision, recall, time-to-detect, false-positive rate, and latency.
  5. Compute precision, recall, time-to-detect, false-positive rate, and latency.
  6. Store the full run (inputs, verdicts, timings) so it is replayable.

Publishing​

Runs are generated automatically by the autonomous scheduler on the configured benchmark cadence. A result becomes public only after human review (the same human-approval discipline we apply before anything is published). The latest approved numbers are served from /api/public/benchmark/summary and surfaced on the Dralvia homepage. Until a run is approved, no accuracy figure is shown.

A run is only eligible for publishing when it is representative. It must include enough genuinely-live phishing pages (a precision figure drawn from a handful of detections is statistical noise) and must clear minimum precision and recall floors. When a run does not yet have enough live ground truth, the public pages honestly show "measurement in progress" rather than a misleading number. This is a limit of how much fresh, live ground truth a given run captured, not a detection failure.

Quick and deeper scan tracks have separate publication status. The public headline is the default quick track because it reflects the fast user flow, but a deeper track can be shown alongside it when available. A fast result is not allowed to borrow the deeper track's recall, and a deeper result is not allowed to borrow the quick track's latency. If either track misses the publication floors, that track is held back until a representative run clears the gates.

The scheduler writes candidate run artifacts and status into the internal benchmark runtime. Public pages read only the approved published artifact. Auto publish is disabled by default; enabling it still keeps the same sanity gates in front of the public number.

What the pending state tells you​

Until a result is approved, the Detection Quality page states exactly where the pipeline is, live from /api/public/benchmark/summary:

  • Collecting live samples: the latest automated run confirmed fewer live phishing samples than the publication bar, with the current count and the bar shown.
  • Awaiting review: a run that meets the sample bar has completed and is waiting on human approval.
  • Not representative yet: the latest run did not meet the publication bar, so its numbers are withheld and the next run replaces it.
  • No run yet: no automated run has been recorded on the deployment.

The page also states the run cadence, so you know when the state can next change. Unpublished accuracy numbers are never exposed in the pending state; they appear only after approval.

Honest limits​

  • These are point-in-time measurements on the sampled fresh ground truth, not a guarantee of future detection on every possible threat.
  • Time-to-detect depends on how quickly a URL appears in a public feed; we report the delta from feed first-seen, which is the part we can measure.