Skip to content
BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

219 Commits
docs
docs
 
 
pic
pic
 
 
PAPERS_CLASSIFIED.md
PAPERS_CLASSIFIED.md
 
 
PAPERS_CLASSIFIED_zh.md
PAPERS_CLASSIFIED_zh.md
 
 
README.md
README.md
 
 
README_zh.md
README_zh.md
 
 

Repository files navigation

LLM4Pentest: LLM-Powered Automated Penetration Testing

Lab - ChengHuang

English | 中文

A curated list of resources about the application of Large Language Models in automated penetration testing.
This repository collects relevant academic papers, technical blogs, code, practical tools, and evaluation benchmarks.

Contributions via Pull Requests are welcome!

📄 Based on the paper: Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing

🔬 Experiment Results: https://simon-p-j-r.github.io/LLM4Pentest/

The online page presents our experiment results for the DeepSeek-v3.2 and DeepSeek-v4 models. The DeepSeek-v3.2 tables cover representative frameworks and agents such as ctfSolver, LuaN1aoAgent, tinyctfer, xbow-competition, Cruiser, CHYing-agent, SickHackShark, newmapta, sub-agent-autopt, CyberStrikeAI, H-Pentest, VulnBot, and PentestGPT, together with comparison baselines baseline-kimi and baseline-cc.

The DeepSeek-v4 tables include representative frameworks such as CAIRN, AWE, OpenClaw, CAI, ctfagent, and Cochise, together with comparison baselines baseline-kimi and baseline-cc.

LLM-Based AutoPT Taxonomy and Framework

Table of Contents

Papers

Academic Papers on LLMs for Penetration Testing, Vulnerability Mining, and Security Analysis

Classification rule: labels reflect the paper's principal research contribution—not merely the environment used in its experiments. Papers spanning multiple areas are assigned to the category that best matches their stated primary contribution.

Label Legend

Label Category Inclusion criterion Count
🤖 SYSTEM & AGENT Systems & Agent Frameworks Runnable offensive-security systems, agent architectures, training methods, planning/reasoning mechanisms, and tool orchestration frameworks. 50
🧪 BENCHMARK & RANGE Benchmarks, Datasets, CTFs & Cyber Ranges Reusable benchmarks, datasets, task suites, cyber ranges, gyms, and evaluation infrastructure. 19
📊 EMPIRICAL EVALUATION Empirical Evaluation & Capability Analysis Studies that measure, compare, or explain offensive capabilities, limitations, autonomy, architecture effects, or risks. 20
📚 SURVEY & MODELING Surveys, Taxonomies & Theoretical Modeling Surveys, SoKs, taxonomies, unified models, and systematic studies of existing research or environments. 10
🛡️ DEFENSE & GOVERNANCE Defense, Remediation, Deception, Ethics & Governance Research centered on defenses against agents, remediation, cyber deception, safeguards, ethics, or research governance. 5
Total 104

  • 📚 SURVEY & MODELING Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing (Link)
    📊 2026 | 🟥 CCF-A | 🏷️ ICLR | 📋 Code: code

  • 🧪 BENCHMARK & RANGE CTFusion: A CTF-based Benchmark for LLM Agent Evaluation (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🧪 BENCHMARK & RANGE CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT Red-MIRROR: Agentic LLM-based Autonomous Penetration Testing with Reflective Verification and Knowledge-augmented Interaction (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT Intelligent Penetration Testing Through Integrated Knowledge Graph and Historical Decision Enhancement (Link)
    📊 2026 | 🟥 CCF-A | 🏷️ IEEE TDSC | 📋 Code: -

  • 🤖 SYSTEM & AGENT Pen-Strategist: A Reasoning Framework for Penetration Testing Strategy Formation and Analysis (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT From Intent to Invocation: A Reasoning-First Framework for Natural Language to Penetration Testing Commands (Link)
    📊 2026 | 🟩 CCF-B | 🏷️ ICASSP | 📋 Code: -

  • 🤖 SYSTEM & AGENT PenForge: On-the-Fly Expert Agent Construction for Automated Penetration Testing (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ ICSE-NIER | 📋 Code: -

  • 🤖 SYSTEM & AGENT Post-Training Local LLM Agents for Linux Privilege Escalation with Verifiable Rewards (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT CHIMERA: Harnessing Multi-Agent LLMs for Automatic Insider Threat Simulation (Link)
    📊 2026 | 🟥 CCF-A | 🏷️ NDSS | 📋 Code: -

  • 🤖 SYSTEM & AGENT WiFiPenTester: Advancing Wireless Ethical Hacking with Governed GenAI (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION AI-Driven Penetration Testing for ARM Systems: Experimental Evaluation and Deployment Framework Across Four Paradigms (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ IEEE Access | 📋 Code: -

  • 🤖 SYSTEM & AGENT CTFAgent: An LLM-Powered Agent for CTF Challenge Solving (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ JISA | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark (Link)
    📊 2026 | 🟥 CCF-A | 🏷️ AAAI | 📋 Code: code

  • 📊 EMPIRICAL EVALUATION Can AI Lower the Barrier to Cybersecurity? A Human-Centered Mixed-Methods Study of Novice CTF Learning (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT STRIATUM-CTF: A Protocol-Driven Agentic Framework for General-Purpose CTF Solving (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📚 SURVEY & MODELING Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🧪 BENCHMARK & RANGE AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 🧪 BENCHMARK & RANGE ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🧪 BENCHMARK & RANGE RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments (Link)
    📊 2026 | 🟥 CCF-A | 🏷️ ICLR | 📋 Code: code

  • 🤖 SYSTEM & AGENT AWE: Adaptive Agents for Dynamic Web Penetration Testing (Link)
    📊 2026 | 🟥 CCF-A | 🏷️ NDSS | 📋 Code: code

  • 🤖 SYSTEM & AGENT FirmAgent: Leveraging Fuzzing to Assist LLM Agents with IoT Firmware Vulnerability Discovery (Link)
    📊 2026 | 🟥 CCF-A | 🏷️ NDSS | 📋 Code: -

  • 🧪 BENCHMARK & RANGE CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Link)
    📊 2026 | 🟥 CCF-A | 🏷️ ICLR | 📋 Code: code

  • 🧪 BENCHMARK & RANGE PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities (Link)
    📊 2026 | 🟥 CCF-A | 🏷️ ICLR | 📋 Code: code

  • 🧪 BENCHMARK & RANGE HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities (Link)
    📊 2026 | 🟥 CCF-A | 🏷️ ICLR | 📋 Code: code

  • 🤖 SYSTEM & AGENT Cyber-Zero: Training Cybersecurity Agents without Runtime (Link)
    📊 2026 | 🟥 CCF-A | 🏷️ ICLR | 📋 Code: code

  • 📊 EMPIRICAL EVALUATION What Makes a Good LLM Agent for Real-world Penetration Testing? (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT PTFusion: LLM-driven Context-aware Knowledge Fusion for Web Penetration Testing (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ Information Fusion | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ Empirical Software Engineering | 📋 Code: -

  • 🤖 SYSTEM & AGENT ZERO-APT: A Closed-Loop Adversarial Framework for LLM-Driven Automated Penetration Testing under Intelligent Defense (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT APT-Agent: Automated Penetration Testing using Large Language Models (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT Automation-Exploit: A Multi-Agent LLM Framework for Adaptive Offensive Security with Digital Twin-Based Risk-Mitigated Exploitation (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION Towards Optimal Agentic Architectures for Offensive Security Tasks (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION Autonomous LLM Agents & CTFs: A Second Look (Link)
    📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🧪 BENCHMARK & RANGE AutoPenBench: A Vulnerability Testing Benchmark for Generative Agents (Link)
    📊 2025 | 🟩 CCF-B | 🏷️ EMNLP Industry Track | 📋 Code: code

  • 🧪 BENCHMARK & RANGE CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities (Link)
    📊 2025 | 🟥 CCF-A | 🏷️ ICML | 📋 Code: code

  • 🤖 SYSTEM & AGENT PenTest++: Elevating Ethical Hacking with AI and Automation (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT Guided Reasoning in LLM-Driven Penetration Testing Using Structured Attack Trees (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ COLM | 📋 Code: -

  • 🤖 SYSTEM & AGENT Automated Penetration Testing: Formalization and Realization (Link)
    📊 2025 | 🟩 CCF-B | 🏷️ Computers & Security | 📋 Code: -

  • 🤖 SYSTEM & AGENT Controller Makes Pentesting Better: An Improved Multi-Agent Automated Penetration Testing Framework (Link)
    📊 2025 | 🟦 CCF-C | 🏷️ TrustCom | 📋 Code: -

  • 🤖 SYSTEM & AGENT PentestMCP: LLM and MCP Based Multi-Agent Framework for Automated Penetration Testing (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ Research Square | 📋 Code: -

  • 🤖 SYSTEM & AGENT Automated Tactics Planning for Cyber Attack and Defense Based on Large Language Model Agents (Link)
    📊 2025 | 🟩 CCF-B | 🏷️ Neural Networks | 📋 Code: -

  • 🤖 SYSTEM & AGENT Perses: Unlocking Privilege Escalation for Small LLMs via Extensible Heterogeneity (Link)
    📊 2025 | 🟦 CCF-C | 🏷️ AsiaCCS | 📋 Code: -

  • 🤖 SYSTEM & AGENT D-CIPHER: Dynamic Collaborative Intelligent Agents with Planning and Heterogeneous Execution for Enhanced Reasoning in Offensive Security (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 🛡️ DEFENSE & GOVERNANCE Poster: Towards Intelligent Assurance for Autonomous AI Pentesters: Concurrent Compliance Auditing and Self-Augmentation via Execution Trace Analysis (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ ACM CCS Poster | 📋 Code: -

  • 📚 SURVEY & MODELING Benchmarking Practices in LLM-Driven Offensive Security: Testbeds, Metrics, and Experiment Design (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📚 SURVEY & MODELING Large Language Models for Cyber Security: A Systematic Literature Review (Link)
    📊 2025 | 🟥 CCF-A | 🏷️ ACM TOSEM | 📋 Code: -

  • 🤖 SYSTEM & AGENT AutoPentester: An LLM Agent-based Framework for Automated Pentesting (Link)
    📊 2025 | 🟦 CCF-C | 🏷️ TrustCom | 📋 Code: code

  • 📚 SURVEY & MODELING A Survey on Agentic Security: Applications, Threats and Defenses (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🛡️ DEFENSE & GOVERNANCE Ethics Statements in Autonomous Penetration-Testing Agent Research (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📚 SURVEY & MODELING Forewarned is Forearmed: A Survey on Large Language Model-based Agents in Autonomous Cyberattacks (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION From Capabilities to Performance: Evaluating Key Functional Properties of LLM Architectures in Penetration Testing (Link)
    📊 2025 | 🟩 CCF-B | 🏷️ EMNLP | 📋 Code: -

  • 🛡️ DEFENSE & GOVERNANCE Cloak, Honey, Trap: Proactive Defenses Against LLM Agents (Link)
    📊 2025 | 🟥 CCF-A | 🏷️ USENIX | 📋 Code: code

  • 🤖 SYSTEM & AGENT EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities (Link)
    📊 2025 | 🟥 CCF-A | 🏷️ ICML | 📋 Code: code

  • 🧪 BENCHMARK & RANGE Shell or Nothing: Real-World Benchmarks and Memory-Activated Agents for Automated Penetration Testing (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT Automating web application vulnerability detection: a generative AI and security tool based penetration testing framework (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ BRAC University | 📋 Code: -

  • 🤖 SYSTEM & AGENT Multi-Agent Penetration Testing AI for the Web (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🛡️ DEFENSE & GOVERNANCE Perry: A High-level Framework for Accelerating Cyber Deception Experimentation (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 🤖 SYSTEM & AGENT Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks (Link)
    📊 2025 | 🟥 CCF-A | 🏷️ TOSEM | 📋 Code: code

  • 🧪 BENCHMARK & RANGE Towards Automated Penetration Testing: Introducing LLM Benchmark, Analysis, and Improvements (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ UMAP | 📋 Code: code

  • 🤖 SYSTEM & AGENT RedTeamLLM: An Agentic AI Framework for Offensive Security (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 🧪 BENCHMARK & RANGE PentestEval: Benchmarking LLM-based Penetration Testing with Modular and Stage-Level Design (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT VulnBot: Autonomous Penetration Testing for a Multi-Agent Collaborative Framework (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 📚 SURVEY & MODELING A Unified Modeling Framework for Automated Penetration Testing (Link)
    📊 2025 | 🟩 CCF-B | 🏷️ Computers & Security | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges (Link)
    📊 2025 | 🟥 CCF-A | 🏷️ ACM CCS | 📋 Code: -

  • 🤖 SYSTEM & AGENT RefPentester: A Knowledge-Informed Self-Reflective Penetration Testing Framework Based on Large Language Models (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 🤖 SYSTEM & AGENT PentestAgent: Incorporating LLM Agents to Automated Penetration Testing (Link)
    📊 2025 | 🟦 CCF-C | 🏷️ AsiaCCS | 📋 Code: code

  • 🤖 SYSTEM & AGENT CAI: An Open, Bug Bounty-Ready Cybersecurity AI (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 🤖 SYSTEM & AGENT Pentest-R1: Towards Autonomous Penetration Testing Reasoning Optimized via Two-Stage Reinforcement Learning (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 📊 EMPIRICAL EVALUATION On the Surprising Efficacy of LLMs for Penetration-Testing (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 🤖 SYSTEM & AGENT Automated Penetration Testing with LLM Agents and Classical Planning (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT xOffense: An AI-driven Autonomous Penetration Testing Framework with Offensive Knowledge-enhanced LLMs and Multi-Agent Systems (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT RapidPen: Fully Automated IP-to-Shell Penetration Testing with LLM-based Agents (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT Construction and Evaluation of LLM-based agents for Semi-Autonomous penetration testing (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 🤖 SYSTEM & AGENT ARACNE: An LLM-Based Autonomous Shell Pentesting Agent (Link)
    📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🧪 BENCHMARK & RANGE Got Root? A Linux Priv-Esc Benchmark (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🧪 BENCHMARK & RANGE Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 📊 EMPIRICAL EVALUATION LLM Agents Can Autonomously Hack Websites (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT ChainReactor: Automated Privilege Escalation Chain Discovery via AI Planning (Link)
    📊 2024 | 🟥 CCF-A | 🏷️ USENIX Security | 📋 Code: -

  • 🤖 SYSTEM & AGENT CIPHER: Cybersecurity Intelligent Penetration-Testing Helper for Ethical Researcher (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ Sensors | 📋 Code: code

  • 🧪 BENCHMARK & RANGE AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION LLM Agents can Autonomously Exploit One-day Vulnerabilities (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT BreachSeek: A Multi-Agent Automated Penetration Tester (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🧪 BENCHMARK & RANGE Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 📚 SURVEY & MODELING SoK: A Comparison of Autonomous Penetration Testing Agents (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ ARES | 📋 Code: -

  • 🤖 SYSTEM & AGENT HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

  • 📊 EMPIRICAL EVALUATION An Empirical Evaluation of LLMs for Solving Offensive Security Challenges (Link)
    📊 2024 | 🟥 CCF-A | 🏷️ NeurIPS | 📋 Code: code

  • 🤖 SYSTEM & AGENT PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing (Link)
    📊 2024 | 🟥 CCF-A | 🏷️ USENIX | 📋 Code: code

  • 🧪 BENCHMARK & RANGE NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security (Link)
    📊 2024 | 🟥 CCF-A | 🏷️ NeurIPS | 📋 Code:
    NYU CTF Bench / LLMctfautomation

  • 🤖 SYSTEM & AGENT AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 🤖 SYSTEM & AGENT PENTEST-AI: An LLM-Powered Multi-Agents Framework for Penetration Testing Automation Leveraging MITRE ATT&CK (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ CSR | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION Teams of LLM Agents can Exploit Zero-Day Vulnerabilities (Link)
    📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📚 SURVEY & MODELING Survey on Automated Penetration Testing Technology Research (China, 自动化渗透测试技术研究综述) (Link)
    📊 2024 | 🟥 CCF-A | 🏷️ Journal of Software | 📋 Code: -

  • 🧪 BENCHMARK & RANGE InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback (Link)
    📊 2023 | 🟥 CCF-A | 🏷️ NeurIPS | 📋 Code: code

  • 🛡️ DEFENSE & GOVERNANCE PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediation (Link)
    📊 2023 | 🟥 CCF-A | 🏷️ CCS | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION Getting pwn’d by AI: Penetration Testing with Large Language Models (Link)
    📊 2023 | 🟥 CCF-A | 🏷️ FSE/ESEC | 📋 Code: code

  • 📊 EMPIRICAL EVALUATION Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions (Link)
    📊 2023 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

  • 📊 EMPIRICAL EVALUATION Language Agents as Hackers: Evaluating Cybersecurity Skills with Capture the Flag (Link)
    📊 2023 | 🟥 CCF-A | 🏷️ MASEC@NeurIPS'23 | 📋 Code: -

  • 📚 SURVEY & MODELING An Empirical Survey of Functions and Configurations of Open-Source Capture the Flag (CTF) Environments (Link)
    📊 2023 | 🟦 CCF-C | 🏷️ JNCA | 📋 Code: -

Blogs

In-depth technical blogs and analysis articles from security researchers, teams, or companies.

Code

Implementation code, PoCs, or open-source projects for related research.

💻 PentestGPT

  • Brief Introduction: Official implementation of PentestGPT.
  • Related Paper: PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing
  • GitHub: https://github.com/GreyDGL/PentestGPT
  • Core Highlights: The framework consists of three core modules: the Reasoning Module, the Generation Module, and the Parsing Module. Each module maintains an LLM session with its specific dialogue and context. Users interact seamlessly with PENTESTGPT, where different modules handle different types of messages. This interaction culminates in a final decision, advising the user on the next steps to undertake in the penetration testing process.

💻 GHOSTCREW

💻 PentestAgent

  • Brief Introduction: Official implementation of PentestAgent.
  • Related Paper: PentestAgent: Incorporating LLM Agents to Automated Penetration Testing
  • GitHub: https://github.com/nbshenxm/pentest-agent
  • Core Highlights: The overall framework consists of four main components: a Reconnaissance Agent, a Search Agent, a Planning Agent, and an Execution Agent. These components collaborate seamlessly to automate the three primary stages of penetration testing: intelligence gathering, vulnerability analysis, and exploitation.

💻 VulnBot

  • One-sentence Summary: An LLM-based autonomous penetration testing framework designed to simulate the collaborative workflow of human penetration testing teams through a multi-agent system.
  • Paper: VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework
  • GitHub: https://github.com/KHenryAegis/VulnBot
  • Core Highlights: The framework decomposes tasks into three specialized phases: reconnaissance, scanning, and exploitation. It introduces a Penetration Task Graph (PTG) mechanism to ensure logical task execution and dependency management. The core architecture includes Planner, Memory Retriever, Generator, Executor, and Summarizer modules, enabling role specialization, path planning, and efficient inter-agent communication.

💻 ctfSolver

  • Brief Introduction: Second place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (Developed by the Xi'an Jiaotong University xjtuHunter team).
  • GitHub: https://github.com/passer-W/ctfSolver
  • Core Highlights: This project aims to utilize LLMs to assist or automatically complete Capture The Flag (CTF) competition challenges. It typically includes a closed-loop process of problem analysis, script generation (primarily Python), automatic execution, and error correction. By translating CTF problems into prompts understandable by LLMs, it guides the model to write and execute payloads to obtain Flags, making it suitable for researching LLM capabilities in code generation and logical reasoning within the realm of cybersecurity offense and defense.

💻 LuaN1aoAgent

  • Brief Introduction: Third place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (Developed by the Guangzhou University Fangban BinX team).
  • GitHub: https://github.com/SanMuzZzZz/LuaN1aoAgent
  • Core Highlights: This project (codenamed "Luanniao") adopts a P-E-R (Plan-Execute-Reflect) triune cognitive collaboration architecture, simulating the "planning, execution, reflection" thought cycle of human security experts. Its core innovation lies in the introduction of causal graph reasoning and dynamic graph planning technologies, transforming traditional linear attack scripts into dynamically evolving Directed Acyclic Graphs (DAGs). Based on a rigorous "Evidence-Hypothesis-Verification" logic chain for attack decision-making, it effectively avoids the issues of blind guessing and hallucination common in large models. Furthermore, the framework supports MCP protocol tool integration and a human-machine collaborative mode, allowing experts to intervene in real-time and correct attack paths during the automation process.

💻 tinyctfer

  • Brief Introduction: Fourth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (Developed by ChainReactors Co-Founder and related members).
  • GitHub: https://github.com/chainreactors/tinyctfer
  • Core Highlights: This project builds a lightweight intent runtime environment focused on translating Large Language Model attack intentions into system actions. It introduces a meta-tool design pattern to endow the Agent with more flexible tool orchestration capabilities. As part of the ChainReactors ecosystem, it efficiently controls underlying security tools and implements precise attack decision-making and execution in automated CTF problem-solving and penetration testing scenarios.

💻 NeuroSploit

  • Brief Introduction: Sixth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (Developed by a joint team from Tsinghua University, Southeast University, and National University of Defense Technology).
  • GitHub: https://github.com/Neuro-Sploit
  • Core Highlights: This project is an AI penetration testing research and infrastructure ecosystem jointly built by top universities. Its core feature is the open-sourcing of a complete local firing range simulator, allowing developers to debug and train Agents locally without limitations, breaking through the call restrictions of competition platforms. Additionally, the project includes an automated evaluation framework specifically designed for CTF AI Agents, as well as tools for the correction and in-depth analysis of the authoritative security dataset XBOW.

💻 xbow-competition

  • Brief Introduction: Seventh place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (Developed by NSFOCUS Operation Service BG Advanced Offense and Defense Department).
  • GitHub: https://github.com/m-sec-org/xbow-competition
  • Core Highlights: This project is an automated CTF solving system built upon MCP, designed specifically for the XBOW range. It adopts a Client-Server separation architecture, where ez-xbow-platform-mcp (Server) is responsible for challenge management, knowledge base retrieval, and Kali container scheduling, while kimi-cli-for-xbow (Client) acts as the AI Agent for decision-making and interaction. The system supports a Daemon mode for unattended automatic problem solving, features a built-in knowledge base for 9 types of vulnerabilities including XSS and SQL injection, and can seamlessly invoke local security tools like Nmap and Sqlmap via the MCP protocol, achieving a fully automated process from problem acquisition to Flag submission.

💻 Cruiser

  • Brief Introduction: Eighth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (Developed by the D@wnEdg3 team from City University of Hong Kong & IIE, CAS).
  • GitHub: https://github.com/TJR181/Cruiser_public
  • Core Highlights: This project is positioned as an exploration of CTF Agent implementation, aiming to build an intelligent agent centered on large language models capable of autonomously completing vulnerability mining, path breakthrough, and Flag capture in isolated environments. The project embodies an attempt to combine academic AI security research with practical offense and defense technologies, exploring the possibilities of full-process automated penetration testing.

💻 CHYing-agent

  • Brief Introduction: Ninth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None
  • GitHub: https://github.com/yhy0/CHYing-agent
  • Core Highlights: This project adopts a dual-Agent collaborative architecture based on LangGraph, with a core design featuring two roles: an Advisor and a Lead Attacker. The Lead Attacker is responsible for specific tool invocation and vulnerability exploitation, while the Advisor intervenes at the start of tasks, upon consecutive failures, or during periodic checks to provide global strategic guidance, effectively mitigating large model hallucinations in long-chain attacks. Emphasizing "zero trust," it implements stable and efficient automated CTF problem-solving through dynamic role swapping (e.g., rotating between DeepSeek and MiniMax models) and automated Flag format verification.

💻 SickHackShark

  • Brief Introduction: Tenth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (Developed by the NSFOCUS Sickhack team).
  • GitHub: https://github.com/SickHackPark/SickHackShark
  • Core Highlights: This project constructs an intelligent penetration Agent framework designed to automate CTF challenges or penetration testing tasks via LLMs. The project focuses on utilizing the reasoning and planning capabilities of large models to drive security tools or write attack scripts, typically involving automated processes of task decomposition, environmental interaction, result analysis, and strategy adjustment.

💻 newmapta

  • Brief Introduction: Fifteenth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (Developed by the HUST-Jinyinhu Joint Team).
  • GitHub: https://github.com/HUST-JYHLab/newmapta
  • Core Highlights: This project was developed by the "HUST-Jinyinhu Joint Team," composed of offense and defense experts from Huazhong University of Science and Technology and Wuhan Jinyinhu Laboratory. The team has long been dedicated to network offense/defense and intelligent security technologies. This system aims to explore AI-driven next-generation network security solutions. Its core lies in the deep integration of academic research and practical capabilities, building an LLM-centric agent capable of achieving full automation from information gathering to vulnerability exploitation in real network environments, aiming to explore the application potential and boundaries of intelligent agents in automated penetration testing.

💻 sub-agent-autopt

  • Brief Introduction: Sixteenth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (Developed by YANGXU, Network Information Center, University of Science and Technology of China).
  • GitHub: https://github.com/yyy1mu/sub-agent-autopt
  • Core Highlights: The project adopts a multi-agent collaborative design architecture, sharing task complexity by building multiple sub-agents focused on specific penetration stages. It utilizes a master control node for global planning and task distribution, effectively resolving the issue of context loss common in single-model architectures during long-chain attacks, thereby achieving a more precise and logically deep automated penetration testing process.

💻 CyberStrikeAI

  • Brief Introduction: Seventeenth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None
  • GitHub: https://github.com/Ed1s0nZ/CyberStrikeAI
  • Core Highlights: This project is a high-performance automated penetration platform written in Golang, which is relatively rare. Its core advantage lies in native support for the MCP protocol, enabling standardized invocation and orchestration of tools by the agent. The platform features built-in templates for over a hundred security tools covering the full attack chain and supports flexible extension via YAML files. Combined with visual attack link tracking and a multi-model compatible intelligent decision engine, it allows users to drive complex penetration testing tasks and obtain structured security assessment reports through simple natural language interaction.

💻 H-Pentest

  • Brief Introduction: Eighteenth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (Nepnep Team).
  • GitHub: https://github.com/hexian2001/H-Pentest
  • Core Highlights: This project builds an integrated AI penetration testing platform. Utilizing Large Language Models as the core of the intelligent agent, it achieves automated control and decision-making for the penetration testing process. It is capable of autonomous attack path planning, security tool scheduling, and result analysis, aiming to reduce manual participation costs and improve the efficiency and coverage of automated penetration testing.

💻 BUUCTF_Agent

  • Brief Introduction: Twenty-fifth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (College of Smart City, Beijing Union University).
  • GitHub: https://github.com/MuWinds/BUUCTF_Agent
  • Core Highlights: This project constructs a scalable intelligent agent framework focused on CTF scenarios. Its core feature is the "Human-Machine Collaboration" mode, which not only supports the Agent in attempting to solve problems automatically but also allows users to intervene and collaborate with the Agent to conquer complex challenges. It also possesses good plugin-based extensibility to adapt to different types of CTF questions.

💻 AgentNote

  • Brief Introduction: Thirty-fifth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
  • Related Paper: None (Developed by the C1JC Team, Communication University of China).
  • GitHub: https://github.com/C1JC/AgentNote
  • Core Highlights: The core of this project lies in the introduction of a "Process Note" mechanism to address the issue of context forgetting in long-chain penetration testing. Unlike traditional Agents that rely solely on dialogue history, AgentNote maintains a structured dynamic notebook that records asset information, verified vulnerabilities, and attack paths in real-time. This design not only endows the large model with external memory similar to human security researchers, ensuring goal consistency during complex multi-round interactions, but also facilitates tracing attack logic from the "notes," enabling more efficient breakpoint resumption and manual audit reviews.

💻 Shannon

  • Brief Introduction: A fully autonomous AI hacker and pentester designed to find actual exploits in web applications, achieving a 96.15% success rate on the hint-free, source-aware XBOW Benchmark.
  • Related Paper: None (The advanced analysis engine in Shannon Pro is inspired by the LLMDFA paper).
  • GitHub: https://github.com/KeygraphHQ/shannon
  • Core Highlights: The core of this project lies in its multi-agent architecture that seamlessly combines white-box source code analysis with black-box dynamic exploitation. Unlike traditional scanners that only flag potential issues, Shannon autonomously hunts for attack vectors and uses a built-in browser to execute real-world exploits (such as injection attacks and authentication bypasses). This approach ensures concrete proof of vulnerabilities, delivering pentester-grade reports with reproducible, copy-and-paste PoCs to eliminate false positives, while running different vulnerability checks in parallel for highly efficient results.

💻 Cyber-Zero

  • Brief Introduction: The first runtime-free framework for synthesizing high-quality agent trajectories to train cybersecurity LLMs without requiring executable environments. It leverages persona-driven LLM simulation from CTF writeups to generate realistic interaction sequences, enabling open-source models to achieve up to 13.1% absolute performance gains and match frontier proprietary models.
  • Related Paper: Cyber-Zero: Training Cybersecurity Agents without Runtime
  • GitHub: https://github.com/amazon-science/Cyber-Zero
  • Core Highlights: The framework's innovation lies in its persona-driven dual-LLM approach (Player Model and Bash Terminal) that reverse-engineers system behaviors to generate multi-turn, realistic attack sequences. By transforming unstructured writeups into structured training data complete with failed attempts and debugging sessions, it overcomes the fundamental data scarcity in cybersecurity. The resulting model, Cyber-Zero-32B, not only matches top proprietary models but does so with superior cost-effectiveness, effectively democratizing state-of-the-art cybersecurity agent development.

💻 HackWorld

  • Brief Introduction: The first evaluation framework designed to systematically assess Computer-Use Agents' capabilities in exploiting web application vulnerabilities through visual interaction. It exposes agents to 36 curated vulnerable web applications spanning 11 frameworks and 7 languages, revealing that even state-of-the-art CUAs achieve exploitation rates below 12%.
  • Related Paper: HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities
  • GitHub: https://github.com/GUI-Agent/HackWorld
  • Core Highlights: The framework's core contribution is shifting evaluation from sanitized benchmarks to realistic, vulnerable environments. By integrating a full Kali Linux toolset (Burp Suite, DirBuster, Nikto) and supporting multiple observation spaces (screenshots, a11y trees, Set-of-Marks), it provides unprecedented insight into agent behavior. The findings are striking: larger, newer models like Claude-4 underperform compared to Claude-3.7, proving that cybersecurity tasks require strategic reasoning and tool orchestration, not just better perception.

💻 Pentest-R1

  • Brief Introduction: A novel two-stage reinforcement learning framework that optimizes LLM reasoning for autonomous penetration testing. It combines offline RL on a curated dataset of 500+ real-world expert walkthroughs with online RL in interactive CTF environments, achieving state-of-the-art results on Cybench and AutoPenBench.
  • Related Paper: Pentest-R1: Towards Autonomous Penetration Testing Reasoning Optimized via Two-Stage Reinforcement Learning
  • GitHub: https://github.com/KHenryAegis/Pentest-R1
  • Core Highlights: The project's key innovation is its two-stage synergy: offline GRPO training instills foundational attack logic from "Thought-Command-Observation" tuples, while online GRPO refines error correction through live environment interaction. The ablation study proves that both stages are essential for peak performance. Remarkably, fine-tuned from just an 8B model, Pentest-R1 rivals and even surpasses proprietary giants like GPT-4o and Claude, proving that strategic RL can outperform brute-force scaling.

💻 EnIGMA+

  • Brief Introduction: EnIGMA+ is an enhanced evaluation scaffolding designed for cybersecurity agents to tackle Capture The Flag (CTF) challenges. Built upon the SWE-agent framework, it serves as the core agent execution component of the Cyber-Zero pipeline, enabling systematic assessment of how large language models handle complex, multi-step offensive security tasks.
  • Related Paper: Cyber-Zero: Training Cybersecurity Agents Without Runtime
  • GitHub: https://github.com/amazon-science/Cyber-Zero/tree/main/enigma-plus
  • Core Highlights: The project introduces a runtime-free trajectory synthesis approach to democratize cybersecurity agent training. Key features include a centralized YAML-based configuration for seamless integration of diverse LLMs and a standardized evaluation protocol that prioritizes turn-based efficiency over cost. By providing repaired benchmark suites like InterCode-CTF and Cybench, EnIGMA+ ensures a stable and fair environment for measuring agent capabilities in capturing flags and generating accurate exploit trajectories.

💻 EnIGMA

  • Brief Introduction: EnIGMA (Enhanced Interactive Generative Model Agent) is a specialized language model (LM) agent built upon the SWE-agent framework, designed to autonomously solve Capture The Flag (CTF) challenges and tackle offensive cybersecurity tasks.
  • Related Paper: EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities
  • GitHub: https://github.com/SWE-agent/SWE-agent
  • Core Highlights: The framework introducing Interactive Agent Tools (IATs). These novel interfaces allow the agent to run and multitask with interactive command-line utilities—such as debuggers (gdb) and server connections—which previous agents could not handle. Additionally, it integrates a new Summarizer to manage long context windows and formally identifies a hallucination phenomenon termed "soliloquizing" (where the model generates its own fake observations), providing methods to quantify and mitigate it.

💻 ARTEMIS

  • Brief Introduction: Automated Red Teaming Engine with Multi-agent Intelligent Supervision, an autonomous agent from the Stanford Trinity project for automating vulnerability discovery.
  • GitHub: https://github.com/Stanford-Trinity/ARTEMIS
  • Core Highlights: Combines a supervisor service with spawned Codex-based subagents to run offensive-security tasks. It supports benchmark-style CTF runs through YAML configuration, can be deployed locally or in Docker, and includes guidance for OpenAI/OpenRouter model configuration, persistent logs, and repeatable test execution.

💻 Strix

  • Brief Introduction: A fully custom-built multi-agent penetration testing framework that utilizes a tree-based agent architecture.
  • GitHub: https://github.com/usestrix/strix
  • Core Highlights: Features a Root Agent for orchestration and Sub-Agents for independent validation based on vulnerability types and components. It supports parallel execution of the discovery-validation-reporting-remediation chain. Incorporates 40+ built-in vulnerability skill files, uses Jinja2 to embed system prompts for knowledge injection, executes within a Docker Kali Linux sandbox for isolation, provides a terminal TUI visualization based on Textual, and supports multiple model backends via LiteLLM.

💻 PentAGI

  • Brief Introduction: A self-hosted autonomous penetration testing platform built around a multi-agent workflow for complex security testing tasks.
  • GitHub: https://github.com/vxcontrol/PentAGI
  • Core Highlights: Uses specialized agents such as Orchestrator, Researcher, Developer, and Executor to decompose and execute pentesting tasks. It supports multiple LLM providers, 20+ classic pentest tools in Docker/Podman-isolated runtimes, several search engines, pgvector/Neo4j-backed knowledge storage, and an observability stack including Grafana and Langfuse.

💻 Darkmoon

  • Brief Introduction: Open-source (GPL-3.0) autonomous AI pentest platform and MCP host.
  • GitHub: https://github.com/ASCIT31/Dark-Moon
  • Core Highlights: Per-tech offensive sub-agents (GraphQL, Spring, .NET, Node, PHP, Ruby, WordPress) plus Active Directory and Kubernetes coverage. Orchestrates 80+ offensive tools over the MCP protocol and keeps an evidence trail per finding.

💻 CAI (Cybersecurity AI)

  • Brief Introduction: A custom multi-agent framework comprising 20+ specialized red and blue team agents, supporting Parallel, Swarm, Hierarchical, Sequential, and Conditional collaboration modes.
  • GitHub: https://github.com/aliasrobotics/cai
  • Core Highlights: Enables context transfer between agents via a Handoff mechanism. Tools are organized by Kill Chain stages and support the MCP protocol for external tool integration (e.g., Burp Suite). It features multi-layer security protections (Input/Output Guardrails, Unicode homograph detection) and supports 300+ models via LiteLLM.

💻 Pentest-Swarm-AI

  • Brief Introduction: A penetration testing system based on a Stigmergic Swarm architecture, utilizing a Blackboard shared state and Pheromone-decay indirect coordination mechanism without a central planner.
  • GitHub: https://github.com/Armur-Ai/Pentest-Swarm-AI
  • Core Highlights: Contains five autonomous responsive agents (Recon/Classifier/Exploit/Report/Seed). It supports both sequential pipeline and decentralized Swarm modes. Integrates 15+ security tools and features a comprehensive evidence capture and reporting system (screenshots, HTTP files, bounty estimation, multi-platform submission templates).

💻 pentest-copilot

  • Brief Introduction: A fully custom three-tier cascaded agent framework designed for automated pentesting (Main Agent 25 rounds → Subagent 15 rounds/10 mins → Swarm Agent 25 rounds/15 mins).
  • GitHub: https://github.com/bugbasesecurity/pentest-copilot
  • Core Highlights: Includes 16 core tools, 5 Burp integration tools, and 5 orchestration tools covering bash execution, Python scripts, browser automation, and full Burp integration. Supports multi-model competition mechanisms (Swarm/Racer), structured EngagementState for decoupled conversational memory, and an on-demand installation registry for 100+ security capabilities.

💻 Apex

  • Brief Introduction: An AI penetration testing tool built on Vercel AI SDK v6, utilizing a three-layer agent architecture (Harness base class + Specialized Agents + Orchestration tools).
  • GitHub: https://github.com/pensarai/apex
  • Core Highlights: Integrates 35+ tools covering the entire lifecycle. Supports Swarm parallel execution (up to 10 concurrency) and a deterministic three-stage workflow (Attack Surface Discovery → Swarm Penetration → Reporting) with resume capabilities. Features a cross-session persistent memory system, three-layer context compression, and dual interaction modes (OpenTUI + React and Headless CLI).

💻 AIDA

  • Brief Introduction: A platform that transforms LLMs into autonomous pentesting agents via the MCP protocol, executing the complete attack chain within a Docker container.
  • GitHub: https://github.com/Vasco0x4/AIDA
  • Core Highlights: Provides 18 MCP tools and supports a three-level command approval mode (Auto/Semi-auto/Manual). Uses WebSockets for real-time execution result pushes. Features a model-agnostic architecture, automatic credential placeholder replacement, a React Web frontend, and an integrated AI Chat panel. Execution environments include built-in aida-pentest or Exegol.

💻 HackingBuddyGPT

  • Brief Introduction: A minimalist LLM automated pentesting tool developed by TU Wien's academic lab, designed around the concept of implementing automated pentesting in just 50 lines of Python code.
  • GitHub: https://github.com/IPA-Lab/HackingBuddyGPT
  • Core Highlights: Features a pure Python custom architecture (UseCase → AutonomousUseCase → Agent → Capability) with 10 Agents covering Linux/Windows privilege escalation, Web pentesting, and REST API testing. Utilizes a simple ReAct loop, boasts lightweight dependencies, and remains completely model-agnostic.

💻 pentest-ai

  • Brief Introduction: An end-to-end penetration testing CLI tool (available as PyPI package ptai) composed of 18 specialized agents.
  • GitHub: https://github.com/0xSteph/pentest-ai
  • Core Highlights: Uses a heuristic rule engine to automatically route target types (Web/AD/Cloud/Mobile). Wraps 200+ security tools and includes a built-in Playwright browser agent for SPA/JS-rendered pages. Automatically discovers and constructs vulnerability attack chains. Supports HITL REPL interaction, 6 report output formats, and native CI/CD integration.

💻 BugTraceAI

  • Brief Introduction: A privacy-friendly autonomous security scanning framework containing 28 agents, with 14 expert exploitation agents covering vulnerabilities like XSS, SSRF, IDOR, and LFI.
  • GitHub: https://github.com/BugTraceAI/BugTraceAI
  • Core Highlights: Features a progressively upgrading detection model (e.g., XSS goes from single request to CDP validation across 6 levels), multi-role consensus voting for false positive filtering (rate <5%), and semantic deduplication based on LanceDB. Equipped with a React 18 Web console and WebSocket real-time event streaming.

💻 Xalgorix

  • Brief Introduction: A zero-dependency, fully custom Go-based AI penetration agent deployed as a single binary, integrating 85+ security tools.
  • GitHub: https://github.com/xalgord/xalgorix
  • Core Highlights: Adopts a 22-stage methodology encoded in System Prompts, using a Hook system to enforce workflow (completion gating, stuck detection, active Skill Suggester). Features dual-layer page perception (semantic tree @eNN + advanced UI control discovery @xpaNN) and 750+ embedded SKILL.md knowledge files. Provides a full Web UI, Bubbletea TUI, and PDF report generation.

💻 DRAKBEN

  • Brief Introduction: A 100% custom-built AI penetration testing framework featuring a unique three-layer cognitive memory system (Working Memory, Evolutionary Memory, and Cross-session Knowledge Base).
  • GitHub: https://github.com/ahmetdrak/drakben
  • Core Highlights: Incorporates a five-layer reflection system (ReAct loop, confidence scoring, cognitive introspection, SelfRefiningEngine strategy evolution, and hallucination detection). Contains 34 registered tools and uses SQLite + ChromaDB with a 90-day TTL. Equipped with a prompt_toolkit-based bilingual terminal REPL and FastAPI REST API.

💻 AutoPentest

  • Brief Introduction: A LangGraph-based dual-layer reflection black-box penetration framework utilizing a Planner-Replanner-Supervisor architecture.
  • GitHub: https://github.com/JuliusHenke/autopentest
  • Core Highlights: Orchestrates 8 OWASP Top 10 specialized agents via StateGraph. Forces replanning after each execution and dynamically adjusts strategies based on feedback. Equipped with tools like PersistentShell, Playwright browser, and NVD API. Offers optional Pinecone vector database RAG support and uses LangSmith for experiment tracking.

💻 watchtower

  • Brief Introduction: A three-agent penetration testing CLI tool based on LangGraph, LangChain, and LiteLLM, where Planner, Worker, and Analyst roles collaborate via a fixed-loop topology.
  • GitHub: https://github.com/fzn0x/watchtower
  • Core Highlights: Integrates 23 security tools invoked as subprocesses. Uses SQLite for persistent state management, natively supporting scan resumption. Features a built-in interactive tool selector, multi-LLM backend compatibility (OpenAI/Anthropic/Gemini/OpenRouter), and offline PDF report exportation capabilities.

💻 Autonomous Penetration Testing Copilot

  • Brief Introduction: A single-file, purely hand-crafted LLM-driven penetration testing agent consisting of approximately 5,000 lines of Python code.
  • GitHub: https://github.com/Krishcalin/Autonomous-Pen-Testing
  • Core Highlights: Includes 30 tools covering the full lifecycle and supports both Claude tool_use and OpenAI function calling protocols. Features a six-stage discovery and validation pipeline and a cross-tool discovery correlation engine. Supports SSH remote or local invocation, includes a built-in stealth mode, breakpoint recovery, and utilizes pure CLI interaction.

💻 VulnClaw

  • Brief Introduction: A completely self-developed AI penetration testing CLI tool (Python). The AgentCore class orchestrates LLM calls and tool execution, supporting two modes: single-shot penetration (15 rounds) and continuous penetration (100 rounds/cycle x 10 cycles).
  • GitHub: https://github.com/Unclecheng-li/VulnClaw
  • Core Highlights: 20 penetration Skills (7 core + 13 specialized) covering Web/Intranet/Android/CTF/OSINT scenarios, including 138 reference documents. 4 built-in tools + 11 MCP server configurations. The prominent feature is its anti-hallucination mechanism—hypothesis verification reminders, failure history tracking, forced path switching, and a CTF flag verification state machine, effectively preventing LLM hallucinations and infinite loops.

💻 AWE

  • Brief Introduction: A memory-augmented multi-agent Web penetration testing framework, covering 10 vulnerability types (XSS/SQLi/SSTI/XXE/Command Injection/LFI/SSRF/IDOR/Default Credentials/Information Disclosure).
  • Related Paper: AWE: Adaptive Agents for Dynamic Web Penetration Testing
  • GitHub: https://github.com/stuxlabs/AWE
  • Core Highlights: A three-tier orchestration architecture (IntelligentOrchestrator / EnhancedXSSOrchestrator / ConversationalAgent), with each vulnerability subsystem implementing a three-level escalation strategy (Rule Engine → Context Analysis → LLM Adaptive). Dual-layer memory system (short-term session + SQLite persistence), complete reasoning tracking (five-step decision loop). Achieved an 87% XSS success rate and a 66.7% blind SQLi success rate on the XBOW benchmark, reducing token consumption by 98% compared to its predecessor.

💻 CortexAI

  • Brief Introduction: A Node.js-based autonomous penetration testing AI agent driven by an Azure OpenAI (GPT-4o) ReAct reasoning loop.
  • GitHub: https://github.com/theelderemo/cortexai
  • Core Highlights: ToolRegistry + PluginLoader registry pattern achieves complete decoupling of 26+ tools, invoked via OpenAI Function Calling format. Dual-layer page perception featuring Puppeteer dynamic rendering + static HTML regex fallback, supporting SPA route detection. Enterprise-grade project management (SQLite persistence, vulnerability lifecycle New→Confirmed→Remediated, HTTP evidence chain, OWASP/CWE classification). Complete audit chain via AgentLogger.

💻 Rogue

  • Brief Introduction: An LLM penetration testing agent integrated with RAG knowledge enhancement, connecting to four authoritative security knowledge sources: PentestMonkey, CAPEC, OWASP WSTG, and CISA KEV, dynamically filtering relevant CVEs based on the scan context.
  • GitHub: https://github.com/faizann24/rogue
  • Core Highlights: Dual-channel network monitoring with Playwright browser automation + CDP protocol, automatically identifying tech stacks/JS libraries/CMS/API endpoints. Batch iterative planning mechanism, passing execution insights from previous batches to subsequent ones to achieve cross-batch adaptation. An independent LLM acts as a strict vulnerability evaluator, filtering false positives using HackerOne bounty standards. Built-in OWASP 2021 Top 10 baseline check plan.

💻 AI_Pentest

  • Brief Introduction: A five-agent collaborative architecture (CoordinatorAgent + ReconAgent + VulnAgent + ExploitAgent + ReportAgent), with a Python/FastAPI backend and a comprehensive Web Dashboard frontend built with Vue 3 + TDesign.
  • GitHub: https://github.com/yuanweipeifang/AI_Pentest
  • Core Highlights: Integration of 30+ Kali tools. ExploitAgent implements an 800+ line automated Web attack chain (SQL Injection/XSS/Command Injection/Path Traversal/File Upload/Deserialization/XXE/SSRF), supporting multi-step chained exploitation. CoordinatorAgent multi-round collaborative planning + short-circuit convergence mechanism. Rich CLI interactive mode + WebSocket real-time push + multi-granularity progress callbacks.

💻 PentestAI-Google-ADK

  • Brief Introduction: A penetration testing framework based on Google ADK v1.19.0, adopting a three-tier orchestration of Orchestrator + 2 composite Agents (Reconnaissance sequential execution / Vulnerability parallel scanning), totaling 7 sub-agents.
  • GitHub: https://github.com/manishmitra017/Pentest-google-adk-agent
  • Core Highlights: Strictly follows the PTES methodology, with a built-in comprehensive authorization verification system (IP ranges/Domain wildcards/CIDR), and enforces human_approved authorization for exploitation and privilege escalation operations. Strong security design awareness (legal warnings, DoS prohibition, exclusion of production environments). The project is in an early prototype stage; only 4 out of 11 tool functions have real implementations, while the rest are placeholders.

💻 Transilience-AI

  • Brief Introduction: A skill-driven penetration testing framework based on Claude Code's native capabilities, featuring 26 skills covering OWASP Top 10 (100%), OWASP LLM Top 10 (100%), SANS Top 25 CWE (90+%), and 53 attack types.
  • GitHub: https://github.com/transilienceai/communitytools
  • Core Highlights: Three-role separation (Coordinator/Executor/Validator), with the Validator performing blind reviews to prevent confirmation bias. Three-layer file system memory (attack-chain.md / experiments.md / context-injection.md), 3-strike no-progress detection + P4b Reset mechanism. Creative research synthesized from three sources (model knowledge + skill cross-referencing + online research), with a symlink architecture to achieve skill reuse.

💻 CHeaT

  • Brief Introduction: A defensive tool (not a pentesting agent), designed to defend networks against autonomous LLM-driven penetration testing agents by embedding adversarial payloads into network assets.
  • GitHub: https://github.com/Daniel-Ayz/CHeaT
  • Related Paper: Cloak, Honey, Trap: Proactive Defenses Against LLM Agents
  • Core Highlights: Implements 6 defense strategies with 15 payload generation techniques (honeytokens + prompt injection). Tool wrapper mechanism hijacks system tool outputs to inject deceptive payloads when LLM agents execute commands. Zero external dependencies — pure Python standard library. Modular architecture (DefenseDatabase / DefenseCreator / DefenseInstaller). Covers strategies from "no vulnerabilities found" deception to honeypot credentials and counter-attack traps. Evaluated against PentestGPT with 11 CTF virtual machines.

💻 Cairn

  • Brief Introduction: A general-purpose AI state-space search and problem-solving engine, validated first on autonomous penetration testing.
  • GitHub: https://github.com/oritera/Cairn
  • Core Highlights: Built on a Blackboard Architecture utilizing a dynamic fact-intent graph with three primitives (Fact, Intent, Hint). Agent Workers run a unified OODA loop (Observe, Orient, Decide, Act) to dynamically generate and execute tasks without predefined roles or workflows. Coordinates exclusively through shared-board stigmergy, eliminating direct communication and information silos. Achieved 3rd place and was the only team to AK (All Kill, 54/54) the 2nd Tencent Cloud Hackathon AI Penetration Testing Challenge. Zero MCP tools, zero RAG, and zero predefined agent roles. Supports Claude Code, Codex, and Pi backends under an AGPLv3 license.

ℹ️ 一键返回目录

MCP Tools

A collection of existing Model Context Protocol (MCP) servers designed to integrate penetration testing tools with LLM agents. Each integration typically requires the Core Tool and an MCP Server.

Reconnaissance

Scanning & Fuzzing

Exploitation & Post-Exploitation

Comprehensive tool

ℹ️ Back to Top

Benchmarks

Datasets or benchmarks used to evaluate the performance of LLMs on penetration testing tasks.

Product Provider Publicly stated mode Primary scope
XBOW XBOW Autonomous multi-agent evaluation Containerized Web exploitation challenges for testing vulnerability discovery and exploit execution.
CyberBattleSim Microsoft Agent-based attack simulation Simulated enterprise-network attacks covering discovery, credential use, and lateral movement.
Vulhub Vulhub Automated environment deployment Reproducible Docker environments for public vulnerabilities and exploit validation.
DVWA DVWA Human-interactive training environment Training and testing of common Web vulnerabilities in an intentionally vulnerable application.
VulnStack VulnStack Human-interactive enterprise lab Enterprise and Active Directory penetration testing, including privilege escalation and lateral movement.
Pikachu Pikachu Human-interactive web lab Web vulnerability training across injection, upload, RCE, SSRF, and related categories.
OWASP Benchmark OWASP Benchmark Project Automated security-tool evaluation Accuracy evaluation of SAST, DAST, and IAST tools using executable Java test cases.
Hack The Box Hack The Box Human training and enterprise labs Hands-on security training across Web, systems, Active Directory, and cloud environments.
NYU CTF Bench NYU-LLM-CTF LLM-agent evaluation Containerized CTF tasks for evaluating cybersecurity problem-solving agents across six categories.
AutoPenBench AutoPenBench Generative-agent evaluation End-to-end agent evaluation on controlled penetration tasks and real CVEs.
Cybench Cybench LLM and agent evaluation CTF-based evaluation using complete tasks and scoreable subtasks.
HackWorld GUI-Agent GUI-based autonomous agent GUI-based Web exploitation through screenshots, mouse actions, and keyboard input.
GOAD Orange Cyberdefense Automated deployment, human-driven testing Active Directory attack practice covering enumeration, credentials, privilege escalation, and lateral movement.
PACEbench PACEbench Agent-based exploitation evaluation Realistic network exploitation across multi-host, chained, and defense-enabled scenarios.
CyberGym UC Berkeley Agent-based PoC generation PoC generation and executable validation for vulnerabilities in real open-source projects.
CloudGoat Rhino Security Labs Automated deployment, human or agent testing Cloud attack scenarios covering enumeration, credentials, IAM escalation, and attack chains.
PentestEval PentestEval Stage-level LLM evaluation Stage-level evaluation of reconnaissance, attack decisions, exploit generation, and revision.

ℹ️ Back to Top

Citation

Please cite our paper if you find this repository or our paper useful in your work:

@misc{peng2026hackershallucinatorscomprehensiveanalysis,
      title={Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing}, 
      author={Jiaren Peng and Zeqin Li and Chang You and Yan Wang and Hanlin Sun and Xuan Tian and Shuqiao Zhang and Junyi Liu and Jianguo Zhao and Renyang Liu and Haoran Ou and Yuqiang Sun and Jiancheng Zhang and Yutong Jiao and Kunshu Song and Chao Zhang and Fan Shi and Hongda Sun and Rui Yan and Cheng Huang},
      year={2026},
      eprint={2604.05719},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2604.05719}, 
}

Follow Us

DasLab Community

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

325 stars

Watchers

3 watching

Forks

44 forks
Report repository

Releases

Packages

Contributors