Skip to content

Triage rubric

Use this rubric to select two or three findings worth checking in a validation loop and measure the time spent on manual triage. The run observations guide explains which conclusions this limited sample can support. The Validate a finding exercise applies the rubric to a first discovery pass.

Budget: ≤ 10 minutes per finding, ~3 hours per tool in total. Go highest tool-reported severity first; stop at the cap or once you have your candidates and a feel for the noise, whichever comes first. If a finding can’t be decided in 10 minutes, mark it needs-investigation and move on.

Field Values Notes
Finding ID tool’s ID or hash stable reference for the full-loop exercise
Tool / version free text include model/backend used
Verdict TPFPneeds-investigationduplicate see decision rules below
Tool severity as reported used for ordering only — no re-rating
Exploitability reachableneeds-preconditionstheoretical TPs only, one line naming what you checked (see exploitability) — this is what picks the loop candidates
Triage time minutes feeds the human-cost picture
Notes free text what decided the verdict and who produced it; for an FP, the reason; assumptions, uncertainties and context needed by another reviewer
  • TP True positive — the flaw exists in the code as described and is a security issue in this codebase’s context (not necessarily exploitable today). A real but minor flaw is still a TP; how urgent it is gets decided in prioritisation.

  • FP False positive — the finding is not a security issue in this codebase. Note one reason:

    • not present: the described flaw is not in the code.
    • unreachable: the flaw is in the code, but no input or configuration the codebase admits reaches it.
    • mitigated: a control elsewhere, such as upstream validation or a framework default, defends the path.
    • quality-only: a code-quality finding with no security relevance.
    • test-only: the code runs only in tests or fixtures.

    The reason keeps hallucinated findings apart from real but defended code, and the first three match the dismissal reasons in GitLab and GitHub, so a row transfers to either platform.

  • needs-investigation — plausible but not decidable within the time box. Not a failure verdict; the rate of these is itself a usability signal.

  • duplicate — same root cause and location as an already-triaged finding (including across tools, when triaging the second tool).

Deployment context decides a verdict where it decides whether attacker input reaches the flaw:

  • configuration and feature flags: FP unreachable, or needs-preconditions;
  • who can send the input, given what the system is for;
  • controls in front of the code: FP mitigated.

How critical the system is, how sensitive its data is and how exposed it is decide how urgent a fix is. Leave those to prioritisation.

Tool reports often state their own confidence: “confirmed”, “verified”, “reproduced”. These rules decide what supports a verdict. They come from studies of automated pipelines and benchmarks, not of people triaging findings, so treat them as cautious defaults.

  1. A tool’s label is not evidence. Base the verdict on evidence you can check yourself, such as the code. Evidence counts only if neither the tool’s own output nor data an attacker controls decided it. Record in Notes what decided the verdict and who produced it. If the code doesn’t decide it within the time box, mark the finding needs-investigation. Agents that check their own work have accepted their own echoed output as proof of success (Lohrasbi et al.). Automated checks that read what the target returns can be forged by whoever controls the target (Fujiyama and Shamim). A test, script or success marker the tool wrote or could edit is its own output too: read what it checks before counting its result. In one study, 51 of 116 proofs of concept that passed their pipeline’s own check never reached the vulnerable code, and some printed the success marker whatever happened (Gezgin et al.). Agents also game tests they can reach: one model passed 54 % of tasks whose tests contradicted the specification (Zhong et al.).
  2. A reproduction needs the real application. Count a reported reproduction only if its artifact comes from the real, unmodified application: a request and the response it produced, a crash, or a callback. A mock server, a harness the tool built, or a re-run of the tool’s own script doesn’t count. In one benchmark, close to half of the agent runs produced such a stand-in, with convincing output that re-ran cleanly (He et al.). The input must also enter where an attacker’s would, through a request, a file or a public API. A proof that calls the vulnerable function directly skips the path that decides reachability (Gezgin et al.). Without a qualifying artifact, triage the finding from the code. A failed reproduction is not evidence against the finding: agents reproduced about one in five bugs known to be real (Wang et al.; Lee et al.).
  3. Trace the path yourself. Agents’ reports cite the flawed lines more reliably than the path from attacker input to them or the point where harm occurs (Li et al.). For a TP, check that path in the code and name it in the Exploitability line.
Value What the line names
reachable The path from attacker input to the flaw, traced in the code
needs-preconditions The path, and the configuration, role or state it depends on
theoretical The flaw, and why you found no attacker path within the time box

A witness shows that a flaw is real. An exploit shows what an attacker gains. A fix decision needs the first.

Evidence Produced by Shows Doesn’t show
Traced path Reading the code, or a data-flow query The flaw is plausibly reachable That the code behaves as read
Witness A failing test, a crash, or a request and its response from the real application, with harmless input The flaw is real and reachable, as tested Impact in production
Counterexample A fuzzer, symbolic execution or a model checker The property fails for one concrete input Anything beyond the inputs explored
Proof Formal verification The property holds or fails for every input, in the model Whether the model matches the deployed system
Exploit Turning the flaw into an attack What an attacker gains More than a fix decision needs

Validation needs a witness. The evidence rules above apply to every row an agent produces. A witness shows behaviour under the tested conditions only, and failing to produce one does not refute the finding.

Exploit development is also the step model providers gate (gated access). The EU Cyber Resilience Act defines an exploitable vulnerability by its “potential to be effectively used by an adversary under practical operational conditions” (Art. 3(41)). Calling a vulnerability exploitable needs no working exploit.

  1. Export the tool’s findings (one row each).
  2. Triage in the tool’s reported-severity order, highest first, until the cap.
  3. Triage solo; findings that stay needs-investigation are candidates for a second look with a colleague.
  4. Fill in the run observations row for the run, and note your 2–3 “would most want proven” candidates.