RedLatentEvidence before autonomy.
← Explore the research

Lab idea / AI-enabled offense

Does the advantage survive a held-out task?

Separate workflow development from evaluation when comparing offensive-security assistants.

RedLatent editorial
Published 2026-10-26 · Updated 2026-10-26 UTC

Untested hypothesis — not a reported experiment

Falsifiable hypothesis

A workflow tuned on a development split retains its advantage on a fixed held-out selection under equivalent conditions.

Source-led motivation

CTF-based evaluation framework with task environments, evaluators, unguided and subtask-guided modes, and per-run logs.

[cybench]

Dockerized CSAW challenges for LLM-agent evaluation spanning web, binary exploitation, forensics, reverse engineering, cryptography, and miscellaneous tasks, with separate development and test sets.

[nyu-ctf]

Code and data for evaluating agents against real web-application CVEs in Docker-based environments, with zero-day and one-day information settings and Inspect-based evaluation.

[cve-bench]

Authorized lab scope

Disposable isolated CTF/benchmark environments you are authorized to run; no route to production.

Prerequisites

  • Separate development and test selections with versioned manifests.
  • Equal model/tool budgets and the same grader revision.
  • Run logs with secrets removed; network isolation beyond container defaults.

Comparison & control

Freeze the workflow before test runs. Compare the baseline and candidate with the same guidance and vulnerability-description availability.

Observations to collect

  • Task completion supported by grader evidence, not model self-report.
  • Budget exhaustion, incomplete runs, failed environments, and guidance used.
  • Any exposure to public solutions or test answers.

Disconfirming evidence

  • The improvement disappears on held-out tasks or depends on unequal guidance, budgets, or leaked answers.

Stop conditions

  • Stop if any action reaches an unapproved target, real user data, or a non-lab service.
  • Stop on unexpected egress, exposed credentials, or an exhausted predeclared budget; retain a sanitized failure record.

Source bibliography

  1. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk ↗

    Cybench research team / andyzorigin

    Publication date: Date not supplied · Editorial check 2026-10-26 UTC

    Project README / official documentation; source-led editorial check, no execution
  2. NYU CTF Bench ↗

    NYU-LLM-CTF research team

    Publication date: Date not supplied · Editorial check 2026-10-26 UTC

    Project README / official documentation; source-led editorial check, no execution
  3. CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities ↗

    UIUC Kang Lab / CVE-Bench authors

    Publication date: Date not supplied · Editorial check 2026-10-26 UTC

    Project README / official documentation; source-led editorial check, no execution

Continue the thread