Skip to content

1 · First discovery pass

Goal: Run one security review inside a sandbox. Keep the raw findings and a short run report so you can compare later runs.

This exercise covers discovery only. Leave triage, testing whether findings are real vulnerabilities, and fixes for later exercises.

Follow the setup steps in Getting started. Use the appsec-sbx wrapper to import a copy of your pilot repo into a disposable Docker sandbox VM. Check the copy for secrets and unrelated files before importing it. The agent works on that copy.

Before starting the agent, complete the environment self-check.

The provided appsec-sbx wrapper incorporates these principles through VM isolation, filtered imports, network policies and stop/reset commands.

Complete the kill-switch rehearsal before the first discovery pass. It ends with a clean reset. Then import the target and skills, and supply a fresh credential for the review. Record the evidence requested by the self-check. Complete all six checks before running any agent.

Before each later independent pass, including the optional comparisons below:

  1. Export any previous results.
  2. Stop the run and revoke the old credential.
  3. Reset and check the VM.
  4. Import the target and reinstall the skills.

See Starting another review.

Install and configure the tool inside the sandbox, including model access. Getting started uses OpenCode with this playbook’s security-review-repo skill. The review skills guide explains how to install and use it. For other discovery tools, see the tool shortlist.

Choose a provider and an API key or supported subscription seat from Providers and credentials. Use a model your organisation permits for this code. See Choosing a model for help choosing one. Confirm the model in the review tool before the run, and record its exact name.

Allow extra network destinations only when the tool needs them. Record the review tool (harness), model and prompt variant. In later comparisons, change one of these at a time.

Set a budget first. If you have measured a demo run, use its token count and cost to estimate your budget. Account for the size of your repo. Write down the limit and how you will enforce it:

  • API billing: set a spending cap on the key or workspace where supported, or use a fixed prepaid balance. If there is no hard cap, write down a limit for stopping the run.
  • Subscription seat: set a limit in elapsed time or visible token count, and watch the run. Record any usage or rate limit reached. If the account can charge for extra usage, cap that separately.
  • Provided API key: check the spending cap with the key owner before you start. If the key runs out mid-pass, stop and record that outcome.

Follow the first-run steps. Stop after exporting the raw findings; leave triage for a later exercise. If the run would exceed the budget, use the kill switch and record why you stopped.

If the tool shows reasoning traces, follow them during the run or read them afterwards. Watch for:

  • Assumptions without evidence.
  • Skipped checks.
  • How the agent responds to blocked actions.
  • Guesses that the repository is a benchmark.

Note anything that stands out. Does it show up in the agent’s actions or findings? Compare the visible reasoning with tool calls and the code the agent inspected. Use it as supporting evidence when you review the run. If reasoning is unavailable, use the tool-call history and final report.

The supplied skill writes its report to ~/out/findings.md in the VM. Export it with appsec-sbx export, then stop the VM. Keep the raw report unchanged and read it as untrusted text. Record the outcome even if the run fails or stops at the budget limit.

Open the Report Template panel below to add notes, or print it and write by hand. Keep a copy alongside the raw export. If you estimate a number, say how you obtained it.

Report Template
Run date
Repo size (estimated thousands of lines of code, or KLOC)
Tech stack: languages and main frameworks
Review tool (harness) and version
Prompt variant and exact model name
Budget or limit for stopping
Tokens used and API cost, or seat usage/rate limit reached
Elapsed time for the agent run
Total findings and counts by reported severity
Reasoning visible? Notes with short excerpts or trace references
Run completed? If stopped, why?
Run report · Fill in online or print and write by hand.

Add short notes on:

  • Setup problems.
  • Refusals, safeguard actions and model redirects. Count these events and note what triggered them, whichever model you used.
  • Suspected noise, blind spots and anything that surprised you.

These are first impressions. Grading findings comes later.

Check what your prompt excludes before you interpret the export. The supplied review skill filters findings by confidence. It excludes classes such as denial of service (DoS), rate limiting and outdated dependencies. Record these limits separately from gaps you noticed during the run.

Use this report for the discovery pass. Leave triage verdicts for the next stage.

  • Self-hosted model: repeat the pass with a model hosted on approved infrastructure. Keep the review tool and prompt fixed. Record setup effort, throughput and differences in the findings. Check hardware needs first.

  • Larger framework: try a heavier tool from the shortlist and record what setup took. Stay within discovery scope and follow the same sandbox and budget requirements.

  • Adapt the review skill: read the skill’s instructions. Note its confidence threshold, exclusions, vulnerability categories and output format. Write down what your repository needs that the prompt misses. This could be a framework the categories miss or an excluded class you want to review.

    To test an idea, pass extra focus, for example /security-review-repo focus on the payment service. Or copy the skill under a new name and change one part. Use appsec-sbx skills to install the modified skill in the sandbox. Keep the original unchanged so you can compare runs. Record the variant in the run report.

  • A working sandbox and evidence for the six environment checks.
  • A raw findings export, or a record of where the run stopped.
  • A run report with budget, runtime, findings counts and setup notes.

These records help you compare tools and models in later runs. They also show where the setup guidance needs work. The next exercise, Validate a finding, triages these findings and checks one of them with a test.