LLM4Pentest: LLM-Powered Automated Penetration Testing

English | 中文

A curated list of resources about the application of Large Language Models in automated penetration testing.
This repository collects relevant academic papers, technical blogs, code, practical tools, and evaluation benchmarks.

Contributions via Pull Requests are welcome!

📄 Based on the paper: Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing

🔬 Experiment Results: https://simon-p-j-r.github.io/LLM4Pentest/

The online page presents our experiment results for the DeepSeek-v3.2 and DeepSeek-v4 models. The DeepSeek-v3.2 tables cover representative frameworks and agents such as ctfSolver, LuaN1aoAgent, tinyctfer, xbow-competition, Cruiser, CHYing-agent, SickHackShark, newmapta, sub-agent-autopt, CyberStrikeAI, H-Pentest, VulnBot, and PentestGPT, together with comparison baselines baseline-kimi and baseline-cc.

The DeepSeek-v4 tables include representative frameworks such as CAIRN, AWE, OpenClaw, CAI, ctfagent, and Cochise, together with comparison baselines baseline-kimi and baseline-cc.

Table of Contents

• Papers
• Blogs
• Code
• MCP Tools
• Benchmarks
• Citation
• Follow Us

Papers

Academic Papers on LLMs for Penetration Testing, Vulnerability Mining, and Security Analysis

• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? (Link)
  📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments (Link)
  📊 2026 | 🟥 CCF-A | 🏷️ ICLR | 📋 Code: code

• AWE: Adaptive Agents for Dynamic Web Penetration Testing (Link)
  📊 2026 | 🟥 CCF-A | 🏷️ NDSS | 📋 Code: code

• CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Link)
  📊 2026 | 🟥 CCF-A | 🏷️ ICLR | 📋 Code: code

• PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities (Link)
  📊 2026 | 🟥 CCF-A | 🏷️ ICLR | 📋 Code: code

• HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities (Link)
  📊 2026 | 🟥 CCF-A | 🏷️ ICLR | 📋 Code: code

• Cyber-Zero: Training Cybersecurity Agents without Runtime (Link)
  📊 2026 | 🟥 CCF-A | 🏷️ ICLR | 📋 Code: code

• What Makes a Good LLM Agent for Real-world Penetration Testing? (Link)
  📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios (Link)
  📊 2026 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• PTFusion: LLM-driven Context-aware Knowledge Fusion for Web Penetration Testing (Link)
  📊 2026 | ⬜ CCF-None | 🏷️ Information Fusion | 📋 Code: -

• LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks (Link)
  📊 2026 | ⬜ CCF-None | 🏷️ Empirical Software Engineering | 📋 Code: -

• From Capabilities to Performance: Evaluating Key Functional Properties of LLM Architectures in Penetration Testing (Link)
  📊 2025 | 🟩 CCF-B | 🏷️ EMNLP | 📋 Code: -

• Cloak, Honey, Trap: Proactive Defenses Against LLM Agents (Link)
  📊 2025 | 🟥 CCF-A | 🏷️ USENIX | 📋 Code: code

• EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities (Link)
  📊 2025 | 🟥 CCF-A | 🏷️ ICML | 📋 Code: code

• Shell or Nothing: Real-World Benchmarks and Memory-Activated Agents for Automated Penetration Testing (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• Automating web application vulnerability detection: a generative AI and security tool based penetration testing framework (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ BRAC University | 📋 Code: -

• Multi-Agent Penetration Testing AI for the Web (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• Perry: A High-level Framework for Accelerating Cyber Deception Experimentation (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

• Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks (Link)
  📊 2025 | 🟥 CCF-A | 🏷️ TOSEM | 📋 Code: code

• Towards Automated Penetration Testing: Introducing LLM Benchmark, Analysis, and Improvements (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ UMAP | 📋 Code: code

• RedTeamLLM: an Agentic AI Framework for Offensive Security (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

• PentestEval: Benchmarking LLM-based Penetration Testing with Modular and Stage-Level Design (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• VulnBot: Autonomous Penetration Testing for a Multi-Agent Collaborative Framework (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

• A Unified Modeling Framework for Automated Penetration Testing (Link)
  📊 2025 | 🟩 CCF-B | 🏷️ Computers & Security | 📋 Code: -

• Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges (Link)
  📊 2025 | 🟥 CCF-A | 🏷️ ACM CCS | 📋 Code: -

• RefPentester: A Knowledge-Informed Self-Reflective Penetration Testing Framework Based on Large Language Models (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

• PentestAgent: Incorporating LLM Agents to Automated Penetration Testing (Link)
  📊 2025 | 🟦 CCF-C | 🏷️ AsiaCCS | 📋 Code: code

• CAI: An Open, Bug Bounty-Ready Cybersecurity AI (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

• Pentest-R1: Towards Autonomous Penetration Testing Reasoning Optimized via Two-Stage Reinforcement Learning (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

• On the Surprising Efficacy of LLMs for Penetration-Testing (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

• Automated Penetration Testing with LLM Agents and Classical Planning (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• xOffense: An AI-driven Autonomous Penetration Testing Framework with Offensive Knowledge-enhanced LLMs and Multi-Agent Systems (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• RapidPen: Fully Automated IP-to-Shell Penetration Testing with LLM-based Agents (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

• ARACNE: An LLM-Based Autonomous Shell Pentesting Agent (Link)
  📊 2025 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? (Link)
  📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• LLM Agents can Autonomously Exploit One-day Vulnerabilities (Link)
  📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• BreachSeek: A Multi-Agent Automated Penetration Tester (Link)
  📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (Link)
  📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

• SoK: A Comparison of Autonomous Penetration Testing Agents (Link)
  📊 2024 | ⬜ CCF-None | 🏷️ ARES | 📋 Code: -

• HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing (Link)
  📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: code

• An Empirical Evaluation of LLMs for Solving Offensive Security Challenges (Link)
  📊 2024 | 🟥 CCF-A | 🏷️ NeurIPS | 📋 Code: code

• PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing (Link)
  📊 2024 | 🟥 CCF-A | 🏷️ USENIX | 📋 Code: code

• NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security (Link)
  📊 2024 | 🟥 CCF-A | 🏷️ NeurIPS | 📋 Code:
  NYU CTF Bench / LLMctfautomation

• AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks (Link)
  📊 2024 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• PENTEST-AI: An LLM-Powered Multi-Agents Framework for Penetration Testing Automation Leveraging MITRE ATT&CK (Link)
  📊 2024 | ⬜ CCF-None | 🏷️ CSR | 📋 Code: -

• PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediation (Link)
  📊 2023 | 🟥 CCF-A | 🏷️ CCS | 📋 Code: -

• Getting pwn’d by AI: Penetration Testing with Large Language Models (Link)
  📊 2023 | 🟥 CCF-A | 🏷️ FSE/ESEC | 📋 Code: code

• Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions (Link)
  📊 2023 | ⬜ CCF-None | 🏷️ arXiv | 📋 Code: -

• Language Agents as Hackers: Evaluating Cybersecurity Skills with Capture the Flag (Link)
  📊 2023 | 🟥 CCF-A | 🏷️ MASEC@NeurIPS'23 | 📋 Code: -

• An Empirical Survey of Functions and Configurations of Open-Source Capture the Flag (CTF) Environments (Link)
  📊 2023 | 🟦 CCF-C | 🏷️ JNCA | 📋 Code: -

ℹ️ Back to Top

Blogs

In-depth technical blogs and analysis articles from security researchers, teams, or companies.

• Penetration Testing with AI – Part 3 (BHIS)

  • Demonstrates the practical application of LLMs in the scanning, exploitation, and report generation phases, covering how AI assists in information gathering, attack surface analysis, and results summarization.

• Generative AI for Offensive Security

  • Introduces the application of Generative AI in Breach and Attack Simulation (BAS) and penetration testing workflows, focusing on how to achieve continuous attack surface management and threat discovery.

• Using AI for Offensive Security

  • Explores the integration methods of AI in penetration testing platforms, including intelligent task allocation, assisted vulnerability analysis, and report generation workflows.

• Offensive AI Use Cases

  • Discusses application scenarios and potential risks of AI in attack simulation, red teaming automation, and adversarial testing.

• Top AI Pentesting Tools 2025

  • Summarizes mainstream AI-assisted penetration testing tools for 2025, analyzing their functional positioning, usage scenarios, and security recommendations.

• AI Pentesting Methodology

  • Introduces the methodology of its AI penetration testing tool, RidgeBot, showing how AI automatically executes the complete penetration testing lifecycle of reconnaissance, vulnerability mining, exploitation, and lateral movement.

• GenAI & Autonomous Security Testing

  • Discusses why autonomous security testing (such as its NodeZero platform) becomes even more critical in the GenAI era, emphasizing the necessity of AI automation platforms in countering AI-driven threats.

• AI PenTesting Governance

  • Analyzes governance and control issues in AI penetration testing, focusing on authorization boundaries, model decision transparency, and audit requirements.

• AI vs Human Expertise

  • Explores the role differences between AI and human experts in penetration testing, analyzing future trends in automation and artificial intelligence collaboration.

• How AI will Impact Automated Pentesting

  • Outlooks how AI will reshape the future of automated penetration testing, discussing its advantages over traditional methods in terms of speed, scale, and identifying complex threats.

• The Fog and Engineering Practice of AI Automated Penetration Testing — Demystifying XBOW

  • Deeply analyzes the technical architecture and engineering practice of XBOW, an AI automated penetration testing platform founded by the former GitHub security team. The article reveals how XBOW solves the fundamental problem of high false positive rates of large models in vulnerability mining through a "Human-Machine Collaboration" hybrid architecture (i.e., LLMs responsible for creative attack exploration, deterministic code responsible for strict verification).

• Automate PenTesting Pipeline

  • Explains how to integrate LLMs/AI agents into the DevSecOps pipeline to achieve automated penetration testing and security verification, including permission control and audit log schemes.

• Introducing advanced tool use on the Claude Developer Platform

  • Anthropic introduces three Beta features: Tool Search (dynamic tool discovery), Programmatic Invocation (code execution to reduce context load), and Usage Examples (improving accuracy), optimizing Claude's tool usage efficiency.

• Top 9 in 7 Days: How I Made Claude Build a Fully Automated CTF Player

  • This project used Claude to develop an automatic CTF tool within 7 days, winning 9th place in the Tencent Cloud Hackathon. The core adopts a dual-Agent collaboration architecture, using a Consultant Agent for review to avoid hallucinations. The tool design is minimalist, with only three key tools.

• AI for Security Attack and Defense: Engineering Design and Practice of Automated Penetration Agent

  • The article proposes the design practice of an AI attack and defense automated penetration Agent, which won 4th place in the Tencent Cloud Hackathon. It innovates with APG structured expert experience and Meta-Tooling code orchestration to solve context explosion and inefficiency problems, achieving efficient attack and defense.

• Code execution with MCP: Building more efficient agents

  • Anthropic proposes using code to replace direct MCP tool calls, loading and filtering data on demand to reduce tokens and improve efficiency, supporting privacy protection and state persistence.

• HexStrike [6.0] AI Automated Penetration Experience

  • Introduces HexStrike AI.

• What to do, I want the Security GPT Agent to work for me every day!

  • Introduces Security GPT.

• AI Large Models Have Upgraded from Cyber Attack Auxiliary Tools to Core Attack Weapons

  • Brief intro: Attackers no longer just use AI tools to improve efficiency, but deeply integrate them into the entire attack chain, deploying AI-enhanced malware with dynamic behaviors.

• US Cyber Command Spends Millions Hiring AI Hackers to Develop Automated Cyber Warfare Weapons

  • Twenty is dedicated to developing offensive AI agents and capabilities for large-scale automated intrusion of foreign targets, having secured a contract worth up to $12.6 million from US Cyber Command. Forbes states this may represent a leap in the US military's automated cyber warfare capabilities.

• [AI Autonomous Penetration Testing Platform] Making Security Testing as Simple as Chatting

  • AI Autonomous Penetration Testing Platform - Built on Golang, with hundreds of built-in security tools, supporting flexible extension of custom tools, and implementing AI intelligent decision-making and automated execution via the MCP protocol.

• Let AI Be Your Cybersecurity Expert! Strix Automatically Discovers Application Vulnerabilities

  • Strix is an AI-based open-source security testing framework equipped with multiple specialized AI agents that collaborate like real hackers to dynamically test your applications and find security vulnerabilities. Unlike traditional scanning tools, Strix not only discovers vulnerabilities but also actually verifies their validity, providing real and reliable security reports.

• An Open Source Global Common Vulnerability Database | Unified Vulnerability Search Platform

  • GlobalCVE is a unified open-source vulnerability intelligence hub dedicated to aggregating and providing vulnerability information worldwide. The project is designed with clarity, collaboration, and security as core concepts, aiming to provide comprehensive vulnerability data for security researchers, developers, and organizations.

• Intelligent Attack Surface Detection Technology and Practice

  • Lingdong Ai.Vul

• CyberStrikeAI — AI Autonomous Penetration Testing Platform

  • Introduction to CyberStrikeAI.

• [AI Automated Penetration] strix Usage Record

  • Strix usage record.

• A High-Performance Vulnerability Scanner Driven by AI -- Deep-Eye

  • Integrates capabilities from multiple AI providers (OpenAI, Grok, OLLAMA, Claude), allowing dynamic switching between providers, and is equipped with comprehensive security testing modules. It features over 45 attack methods, enabling automated vulnerability mining, intelligent payload generation, and professional report output. Supports team-based distributed scanning and features session management capabilities.

• Using AI for Automated Penetration Testing to Decrypt CTF Challenges | CTF Cybersecurity Competition

  • Introduction to AiScan-N.

• EHole (Lengdong) 3.0 Red Team Key Attack System Fingerprint Detection Tool

  • EHole 3.0 (Lengdong 3.0) is an advanced fingerprint detection tool designed for red teams and penetration testers, capable of quickly identifying target system vulnerabilities, service versions, and security weaknesses through various network protocols. It automatically performs system fingerprinting and vulnerability scanning with high stealth, reducing the risk of detection by firewalls and IDS/IPS.

• Ultimate Guide to Penetration Testing Information Gathering: Building a Full-Domain Attack Surface View

  • Systematically expounds on the information gathering phase in penetration testing, the depth and breadth of which directly determine the final effectiveness of the penetration test.

• AI Playing CTF? Build Your Full-Stack Automated Attack and Defense Agent with Cursor + MCP

  • Utilizes the latest MCP (Model Context Protocol), combined with the Cursor editor and Docker containers, to build a fully automated CTF problem-solving agent.

• Which AI Model is Best at Hacking? A Benchmark of 11 LLMs

  • 11 LLMs were benchmarked against 32 black-box cybersecurity hacking challenges to determine which model performs best in offensive security.

• Recommend an Uncensored Model, Especially Suitable for Security Work like Penetration Testing

  • DeepHat (formerly named WhiteRabbitNeo) is an open-source large language model (LLM) specifically designed for cybersecurity. After being renamed in 2024, it became the first truly "uncensored" security AI model in the industry.

• Using MCP to Call Tools for Penetration Testing Is Somewhat Superfluous; There Are Two Reasonable Methods

  • Currently, LLM-powered penetration testing tools can basically be divided into two major schools of thought: those using MCP and those using direct tool calling. The former ranges from Kali MCP to Hexstrike AI, while the latter includes more tools such as the previously tracked PentestGPT, CAI, and the recently popular Strix, Autopentester, etc. Each has its own characteristics and issues.

• Penetration Testing Performance of Hexstrike AI Across Multiple Tools and Models

  • The capability of LLM-assisted penetration testing depends on both tools and models. Finding a suitable match requires multiple attempts and analyses. Hexstrike AI is an MCP-based service; theoretically, all MCP clients and models can be used with it, but its performance varies under different circumstances.

• Completing AI Automated Penetration Testing with Claude Skills

  • Application Ideas of Claude Skills in Automated Penetration Testing

• [Red Team] AI-Powered Autonomous Penetration Testing Platform

  • KaliGPT-Attack Platform is an innovative AI-driven autonomous penetration testing tool that adopts a unique three-module architecture (reasoning, generation, parsing), effectively solving the context loss problem of traditional LLMs during long-term penetration testing processes.

• KaliGPT-Attack Platform: AI-Driven Penetration Testing Technology Implementation and Practical Analysis

  • As an AI-native penetration testing tool, KaliGPT-Attack Platform achieves full-process autonomy and intelligence in penetration testing through innovative architecture design and technology integration, providing an efficient technical solution for the security testing field.

• Use CC Directly: Manus Core Context Technology Has Been Turned into Skills

  • planning-with-files is a Skill that has recently gone viral in the open-source community, garnering 3.3k stars in just four days since its release. It is still growing.

• [RedTeamLLM] Applying Agentic AI in Automated Penetration Testing

  • RedTeamLLM proposes an automated penetration tool framework of "Task Decomposition + Reasoning + Action + Memory," which significantly improves the completion rate and efficiency of automated penetration testing without human intervention.

• OpenAI Sounds "High-Risk" Alarm: CTF Win Rate Soars from 27% to 76%!

  • OpenAI has issued a notable warning: its upcoming AI model is about to reach the "High" level of cybersecurity risk.

• DeepWiki

  • DeepWiki provides conversational, up-to-date documentation for every repository in the world. Think of it as deep research for GitHub.

• WebTrap Park: Web Agent Security Automated Testing Range

  • A Web Agent security automated testing range that requires no modification to existing Web Agents, no email contact, and is ready to use out of the box has officially launched!

• Demystifying the Evaluation of AI Agents

  • The very capabilities that make agents useful also make them difficult to evaluate. Strategies suitable for various deployment environments combine multiple techniques to match the complexity of the systems they evaluate.

• Anthropic Publishes Another Long Article: First Detailed Demystification of the Entire Agent Evaluation Process [Experience Summary from Claude Code Development]

  • Anthropic has summarized a rigorous and practical evaluation method for AI Agents during the development of its star products like Claude Code and through collaboration with cutting-edge customers.

• When Penetration Testing Meets AI: I Ran a Complete Security Test Using Open-Source Tools

  • Strix, an open-source framework for penetration testing using AI.

• A Review of Agentic AI and Cybersecurity: Challenges, Opportunities, and Use Case Prototypes

  • This review aims to explore the impact of agentic AI on cybersecurity. On the defensive side, agent capabilities enable large-scale continuous monitoring, autonomous incident response, adaptive threat hunting, and fraud detection. Conversely, these same characteristics also enhance the offensive side by accelerating reconnaissance, vulnerability exploitation, coordinated operations, and social engineering attacks.

• G.O.S.S.I.P Reading Recommendation 2026-01-16 AI-Driven XSS Security Detection

• Reject the Information Gap, The Most Comprehensive Penetration Testing Cyber Range Recommendations on the Internet [2026 Latest Edition]

  • This time, 7 commonly used cyber range platforms have been added, such as the Burp Suite range, Lingjing range, The Hacker Labs, etc., covering web penetration, SRC vulnerability mining, zero-foundation learning paths, intranet penetration, domain penetration, intranet lateral movement, and more.

• Can Small Security Vertical Models Outperform Large Models? Look at the Results of Israel's Novee

  • Novee's model is only 4B, but it has demonstrated very strong capabilities in actual testing.

• We Used AI to Extract the "Hacker Mindset" from 88,636 Real Vulnerabilities, and Now We Are Opening the Methodology to Everyone

  • Created a Claude Code Skill to enable AI to think like a senior security expert. Behind this is the knowledge accumulation of 88,636 real vulnerability cases.

• New Paradigm in AI Security 3: Theory and Practice of CoT Agents in Automated Penetration Testing

  • The introduction of Chain of Thought (CoT) technology has endowed AI Agents with logical reasoning capabilities. By simulating the "Observe-Orient-Decide-Act" (OODA) Loop of human experts, CoT transforms black-box AI decisions into explainable, logically coherent attack paths.

• We Deconstructed 5,621 XianZhi Research Articles and Wrote a Researcher Skill

  • Went through all 5,621 security research articles from the XianZhi community and turned them into a reusable "Skill", allowing AI to think like a top security researcher.

• Automated Security Testing System Shannon

  • It is not just another vulnerability scanner, but aims to turn the entire penetration testing process into an automated pipeline. The AI agent reads code, runs tools, and operates browsers; ultimately delivering not "potential risks", but "evidence of successful penetration".

• JoySafeter Heavily Open-Sourced: Unlocking a New Paradigm in AI-Driven Security (AISecOps)

  • JoySafeter is an "operating system" for security capabilities; it is not a single tool, but a visual platform capable of uniformly orchestrating disordered security tools and scattered expert experience into a collaboratively fighting AI legion.

• [ICML'25 Paper | Cybersecurity] Can AI Agents Autonomously Launch Cyber Attacks?

  • This article introduces the paper "CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities" published at ICML 2025, from the School of Computing and Data Sciences at the University of Illinois Urbana-Champaign (UIUC). The research team constructed CVE-Bench, the first cybersecurity benchmark based on real-world vulnerabilities, to evaluate the ability of large language model agents to exploit web application vulnerabilities. Experiments show that current state-of-the-art agent frameworks can successfully exploit up to 13% of critical vulnerabilities, a finding that reveals the potential threats posed by AI in the cybersecurity domain.

• DARPA AIxCC Cybersecurity Challenge: How Can AI Autonomously Mine Vulnerabilities and Apply Patches?

  • The first systematic analysis of the DARPA 2023-2025 AI Cyber Challenge (AIxCC), demystifying the technical architecture, competition design, and implementation insights of the world's top Cyber Reasoning Systems (CRS).

• [AI Automated Penetration] BaTianHu (Decepticons) Multi-Agent Automated Penetration Framework

  • An automated red team penetration tool with multi-agent intelligent scheduling written by a Korean developer. It supports self-creating MCPs and the function of turning the entire current project into an MCP, with strong modularity.

• The "King of Involutions" in AI Red Team Tools is Born: Predator v2.3.0 Integrates 15 Major Competitors, Security Circle Exclaims "What are humans needed for?"

  • Just a week after PentAGI's open-sourcing blew up the circle, a team quietly released Predator v2.3.0—an AI penetration testing platform that integrates the core capabilities of 15 top tools such as Shannon, Caldera, and PentAGI.

• When Human Experts Off-Duty: A 7x24 Hour Penetration Documentary of an AI Red Team "Silicon-Based Employee"

  • Cyber attack-defense confrontation has entered the intelligent era, and the limitations of the traditional manual attack-defense mode have become increasingly prominent. As a dimensionality-enhancing striker in cyber attack-defense exercises, AI Red Team agents, relying on large models and hybrid agent technology, have realized the leap of cyber attack-defense from "labor-intensive" to "intelligence-intensive", creating a brand-new intelligent security attack-defense mode.

• The "Strongest" AI Penetration Testing Agent in China? No, I Just Got 4th Place as the Only All-Killed Team – A Retrospective of TCH·Tencent Cloud Hackathon 2nd Intelligent Penetration Challenge

  • A hardcore retrospective of being the "only All-Killed yet only placed 4th": The author spent RMB 7,692 in API costs and used a "anti-multi-agent" minimalist architecture to solve all challenges in the Tencent Cloud Hackathon's 2nd Intelligent Penetration Challenge, but ended up in 4th place due to dynamic scoring rules. This is also the first public release of his "Less Is More" AI penetration system, Cairn.

• Practice of Automated Penetration Testing Tool Development

  • In 2022, the Tide Security Team explored automated penetration testing tools, integrating functions such as asset discovery, service identification, crawling, passive monitoring, vulnerability scanning, POC detection, screenshot capturing, and report generation into a unified system.

• Exploration of AI-based Automated Testing Tools

  • The Tide Security Team proposed the TestFlow automated penetration testing tool and shared its design concepts and methodology.

• Summary and Analysis of the Two Tencent Cloud Hackathons

  • This article systematically reviews the technical solutions of the 20 finalist teams from the first and second Tencent Cloud Intelligent Penetration Hackathons. From the XBow Benchmark to Tencent's self-built cyber range, and from minimalism to control-first approaches, it presents a complete technical landscape of the AI autonomous penetration testing field.

• The First Year of Intelligent Offense and Defense: Penetration Testing Agents Face a Major Test, How AI Evolves from "Capable" to "Controllable"

  • The 2nd Tencent Cloud Hackathon Intelligent Penetration Challenge not only verified the practical potential of AI agents in automated security offense and defense, but also exposed their shortcomings in cognitive common sense regarding real-world scenarios, prompting an industry-wide consensus to shift towards exploring and building a "trusted and controllable" AI security system.

• ThreatBook Flocks — Practicing the Core System of AI Penetration Testing

  • With the deep integration of large model technology and cybersecurity offense and defense, AI penetration testing has upgraded from traditional "manual instruction assistance" to a standardized, automated, and closed-loop intelligent offense and defense system.

• After Two Sessions of TCH — The Harness Engineering Evolution, Defense, and My Reflections on AI Penetration Testing Agents

  • The Harness Engineering Evolution, Defense, and My Reflections on AI Penetration Testing Agents

• AI's Offense and Defense Answer Sheet

  • Throughout May, AI was deeply utilized for enterprise SRC hunting and red team actual combat. From daily vulnerability detection and code auditing to high-intensity, high-confrontation red team projects, relying entirely on DeepSeek v4, a large number of practical, real-world cases and combat experiences were accumulated.

• Redefining Intelligent Penetration: Technical Analysis and Industry Observation of the Excellent TCH Open Source Project CyberStrikeAI

  • Interpretation of CyberStrikeAI

• Paper Intensive Reading: Hackers or Hallucinators? A 10,000-Word Hardcore Deconstruction of LLM Automated Penetration Testing

  • Paper Intensive Reading

  ℹ️ Back to Top

Code

Implementation code, PoCs, or open-source projects for related research.

💻 PentestGPT

• Brief Introduction: Official implementation of PentestGPT.
• Related Paper: PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing
• GitHub: https://github.com/GreyDGL/PentestGPT
• Core Highlights: The framework consists of three core modules: the Reasoning Module, the Generation Module, and the Parsing Module. Each module maintains an LLM session with its specific dialogue and context. Users interact seamlessly with PENTESTGPT, where different modules handle different types of messages. This interaction culminates in a final decision, advising the user on the next steps to undertake in the penetration testing process.

---

💻 GHOSTCREW

• Brief Introduction: An upgraded version of the PentestGPT codebase.
• Related Paper: PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing
• GitHub: https://github.com/GH05TCREW/PentestAgent
• Core Highlights: Features the separation of automated planning and execution. The entire framework is based on PentestGPT, but features upgrades in code composition, tool invocation, and user interaction mechanisms.

---

💻 PentestAgent

• Brief Introduction: Official implementation of PentestAgent.
• Related Paper: PentestAgent: Incorporating LLM Agents to Automated Penetration Testing
• GitHub: https://github.com/nbshenxm/pentest-agent
• Core Highlights: The overall framework consists of four main components: a Reconnaissance Agent, a Search Agent, a Planning Agent, and an Execution Agent. These components collaborate seamlessly to automate the three primary stages of penetration testing: intelligence gathering, vulnerability analysis, and exploitation.

---

💻 VulnBot

• One-sentence Summary: An LLM-based autonomous penetration testing framework designed to simulate the collaborative workflow of human penetration testing teams through a multi-agent system.
• Paper: VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework
• GitHub: https://github.com/KHenryAegis/VulnBot
• Core Highlights: The framework decomposes tasks into three specialized phases: reconnaissance, scanning, and exploitation. It introduces a Penetration Task Graph (PTG) mechanism to ensure logical task execution and dependency management. The core architecture includes Planner, Memory Retriever, Generator, Executor, and Summarizer modules, enabling role specialization, path planning, and efficient inter-agent communication.

---

💻 ctfSolver

• Brief Introduction: Second place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (Developed by the Xi'an Jiaotong University xjtuHunter team).
• GitHub: https://github.com/passer-W/ctfSolver
• Core Highlights: This project aims to utilize LLMs to assist or automatically complete Capture The Flag (CTF) competition challenges. It typically includes a closed-loop process of problem analysis, script generation (primarily Python), automatic execution, and error correction. By translating CTF problems into prompts understandable by LLMs, it guides the model to write and execute payloads to obtain Flags, making it suitable for researching LLM capabilities in code generation and logical reasoning within the realm of cybersecurity offense and defense.

---

💻 LuaN1aoAgent

• Brief Introduction: Third place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (Developed by the Guangzhou University Fangban BinX team).
• GitHub: https://github.com/SanMuzZzZz/LuaN1aoAgent
• Core Highlights: This project (codenamed "Luanniao") adopts a P-E-R (Plan-Execute-Reflect) triune cognitive collaboration architecture, simulating the "planning, execution, reflection" thought cycle of human security experts. Its core innovation lies in the introduction of causal graph reasoning and dynamic graph planning technologies, transforming traditional linear attack scripts into dynamically evolving Directed Acyclic Graphs (DAGs). Based on a rigorous "Evidence-Hypothesis-Verification" logic chain for attack decision-making, it effectively avoids the issues of blind guessing and hallucination common in large models. Furthermore, the framework supports MCP protocol tool integration and a human-machine collaborative mode, allowing experts to intervene in real-time and correct attack paths during the automation process.

---

💻 tinyctfer

• Brief Introduction: Fourth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (Developed by ChainReactors Co-Founder and related members).
• GitHub: https://github.com/chainreactors/tinyctfer
• Core Highlights: This project builds a lightweight intent runtime environment focused on translating Large Language Model attack intentions into system actions. It introduces a meta-tool design pattern to endow the Agent with more flexible tool orchestration capabilities. As part of the ChainReactors ecosystem, it efficiently controls underlying security tools and implements precise attack decision-making and execution in automated CTF problem-solving and penetration testing scenarios.

---

💻 NeuroSploit

• Brief Introduction: Sixth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (Developed by a joint team from Tsinghua University, Southeast University, and National University of Defense Technology).
• GitHub: https://github.com/Neuro-Sploit
• Core Highlights: This project is an AI penetration testing research and infrastructure ecosystem jointly built by top universities. Its core feature is the open-sourcing of a complete local firing range simulator, allowing developers to debug and train Agents locally without limitations, breaking through the call restrictions of competition platforms. Additionally, the project includes an automated evaluation framework specifically designed for CTF AI Agents, as well as tools for the correction and in-depth analysis of the authoritative security dataset XBOW.

---

💻 xbow-competition

• Brief Introduction: Seventh place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (Developed by NSFOCUS Operation Service BG Advanced Offense and Defense Department).
• GitHub: https://github.com/m-sec-org/xbow-competition
• Core Highlights: This project is an automated CTF solving system built upon MCP, designed specifically for the XBOW range. It adopts a Client-Server separation architecture, where ez-xbow-platform-mcp (Server) is responsible for challenge management, knowledge base retrieval, and Kali container scheduling, while kimi-cli-for-xbow (Client) acts as the AI Agent for decision-making and interaction. The system supports a Daemon mode for unattended automatic problem solving, features a built-in knowledge base for 9 types of vulnerabilities including XSS and SQL injection, and can seamlessly invoke local security tools like Nmap and Sqlmap via the MCP protocol, achieving a fully automated process from problem acquisition to Flag submission.

---

💻 Cruiser

• Brief Introduction: Eighth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (Developed by the D@wnEdg3 team from City University of Hong Kong & IIE, CAS).
• GitHub: https://github.com/TJR181/Cruiser_public
• Core Highlights: This project is positioned as an exploration of CTF Agent implementation, aiming to build an intelligent agent centered on large language models capable of autonomously completing vulnerability mining, path breakthrough, and Flag capture in isolated environments. The project embodies an attempt to combine academic AI security research with practical offense and defense technologies, exploring the possibilities of full-process automated penetration testing.

---

💻 CHYing-agent

• Brief Introduction: Ninth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None
• GitHub: https://github.com/yhy0/CHYing-agent
• Core Highlights: This project adopts a dual-Agent collaborative architecture based on LangGraph, with a core design featuring two roles: an Advisor and a Lead Attacker. The Lead Attacker is responsible for specific tool invocation and vulnerability exploitation, while the Advisor intervenes at the start of tasks, upon consecutive failures, or during periodic checks to provide global strategic guidance, effectively mitigating large model hallucinations in long-chain attacks. Emphasizing "zero trust," it implements stable and efficient automated CTF problem-solving through dynamic role swapping (e.g., rotating between DeepSeek and MiniMax models) and automated Flag format verification.

---

💻 SickHackShark

• Brief Introduction: Tenth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (Developed by the NSFOCUS Sickhack team).
• GitHub: https://github.com/SickHackPark/SickHackShark
• Core Highlights: This project constructs an intelligent penetration Agent framework designed to automate CTF challenges or penetration testing tasks via LLMs. The project focuses on utilizing the reasoning and planning capabilities of large models to drive security tools or write attack scripts, typically involving automated processes of task decomposition, environmental interaction, result analysis, and strategy adjustment.

---

💻 newmapta

• Brief Introduction: Fifteenth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (Developed by the HUST-Jinyinhu Joint Team).
• GitHub: https://github.com/HUST-JYHLab/newmapta
• Core Highlights: This project was developed by the "HUST-Jinyinhu Joint Team," composed of offense and defense experts from Huazhong University of Science and Technology and Wuhan Jinyinhu Laboratory. The team has long been dedicated to network offense/defense and intelligent security technologies. This system aims to explore AI-driven next-generation network security solutions. Its core lies in the deep integration of academic research and practical capabilities, building an LLM-centric agent capable of achieving full automation from information gathering to vulnerability exploitation in real network environments, aiming to explore the application potential and boundaries of intelligent agents in automated penetration testing.

---

💻 sub-agent-autopt

• Brief Introduction: Sixteenth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (Developed by YANGXU, Network Information Center, University of Science and Technology of China).
• GitHub: https://github.com/yyy1mu/sub-agent-autopt
• Core Highlights: The project adopts a multi-agent collaborative design architecture, sharing task complexity by building multiple sub-agents focused on specific penetration stages. It utilizes a master control node for global planning and task distribution, effectively resolving the issue of context loss common in single-model architectures during long-chain attacks, thereby achieving a more precise and logically deep automated penetration testing process.

---

💻 CyberStrikeAI

• Brief Introduction: Seventeenth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None
• GitHub: https://github.com/Ed1s0nZ/CyberStrikeAI
• Core Highlights: This project is a high-performance automated penetration platform written in Golang, which is relatively rare. Its core advantage lies in native support for the MCP protocol, enabling standardized invocation and orchestration of tools by the agent. The platform features built-in templates for over a hundred security tools covering the full attack chain and supports flexible extension via YAML files. Combined with visual attack link tracking and a multi-model compatible intelligent decision engine, it allows users to drive complex penetration testing tasks and obtain structured security assessment reports through simple natural language interaction.

---

💻 H-Pentest

• Brief Introduction: Eighteenth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (Nepnep Team).
• GitHub: https://github.com/hexian2001/H-Pentest
• Core Highlights: This project builds an integrated AI penetration testing platform. Utilizing Large Language Models as the core of the intelligent agent, it achieves automated control and decision-making for the penetration testing process. It is capable of autonomous attack path planning, security tool scheduling, and result analysis, aiming to reduce manual participation costs and improve the efficiency and coverage of automated penetration testing.

---

💻 BUUCTF_Agent

• Brief Introduction: Twenty-fifth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (College of Smart City, Beijing Union University).
• GitHub: https://github.com/MuWinds/BUUCTF_Agent
• Core Highlights: This project constructs a scalable intelligent agent framework focused on CTF scenarios. Its core feature is the "Human-Machine Collaboration" mode, which not only supports the Agent in attempting to solve problems automatically but also allows users to intervene and collaborate with the Agent to conquer complex challenges. It also possesses good plugin-based extensibility to adapt to different types of CTF questions.

---

💻 AgentNote

• Brief Introduction: Thirty-fifth place in the Tencent Cloud Hackathon Intelligent Penetration Challenge.
• Related Paper: None (Developed by the C1JC Team, Communication University of China).
• GitHub: https://github.com/C1JC/AgentNote
• Core Highlights: The core of this project lies in the introduction of a "Process Note" mechanism to address the issue of context forgetting in long-chain penetration testing. Unlike traditional Agents that rely solely on dialogue history, AgentNote maintains a structured dynamic notebook that records asset information, verified vulnerabilities, and attack paths in real-time. This design not only endows the large model with external memory similar to human security researchers, ensuring goal consistency during complex multi-round interactions, but also facilitates tracing attack logic from the "notes," enabling more efficient breakpoint resumption and manual audit reviews.

---

💻 Shannon

• Brief Introduction: A fully autonomous AI hacker and pentester designed to find actual exploits in web applications, achieving a 96.15% success rate on the hint-free, source-aware XBOW Benchmark.
• Related Paper: None (The advanced analysis engine in Shannon Pro is inspired by the LLMDFA paper).
• GitHub: https://github.com/KeygraphHQ/shannon
• Core Highlights: The core of this project lies in its multi-agent architecture that seamlessly combines white-box source code analysis with black-box dynamic exploitation. Unlike traditional scanners that only flag potential issues, Shannon autonomously hunts for attack vectors and uses a built-in browser to execute real-world exploits (such as injection attacks and authentication bypasses). This approach ensures concrete proof of vulnerabilities, delivering pentester-grade reports with reproducible, copy-and-paste PoCs to eliminate false positives, while running different vulnerability checks in parallel for highly efficient results.

---

💻 Cyber-Zero

• Brief Introduction: The first runtime-free framework for synthesizing high-quality agent trajectories to train cybersecurity LLMs without requiring executable environments. It leverages persona-driven LLM simulation from CTF writeups to generate realistic interaction sequences, enabling open-source models to achieve up to 13.1% absolute performance gains and match frontier proprietary models.
• Related Paper: Cyber-Zero: Training Cybersecurity Agents without Runtime
• GitHub: https://github.com/amazon-science/Cyber-Zero
• Core Highlights: The framework's innovation lies in its persona-driven dual-LLM approach (Player Model and Bash Terminal) that reverse-engineers system behaviors to generate multi-turn, realistic attack sequences. By transforming unstructured writeups into structured training data complete with failed attempts and debugging sessions, it overcomes the fundamental data scarcity in cybersecurity. The resulting model, Cyber-Zero-32B, not only matches top proprietary models but does so with superior cost-effectiveness, effectively democratizing state-of-the-art cybersecurity agent development.

---

💻 HackWorld

• Brief Introduction: The first evaluation framework designed to systematically assess Computer-Use Agents' capabilities in exploiting web application vulnerabilities through visual interaction. It exposes agents to 36 curated vulnerable web applications spanning 11 frameworks and 7 languages, revealing that even state-of-the-art CUAs achieve exploitation rates below 12%.
• Related Paper: HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities
• GitHub: https://github.com/GUI-Agent/HackWorld
• Core Highlights: The framework's core contribution is shifting evaluation from sanitized benchmarks to realistic, vulnerable environments. By integrating a full Kali Linux toolset (Burp Suite, DirBuster, Nikto) and supporting multiple observation spaces (screenshots, a11y trees, Set-of-Marks), it provides unprecedented insight into agent behavior. The findings are striking: larger, newer models like Claude-4 underperform compared to Claude-3.7, proving that cybersecurity tasks require strategic reasoning and tool orchestration, not just better perception.

---

💻 Pentest-R1

• Brief Introduction: A novel two-stage reinforcement learning framework that optimizes LLM reasoning for autonomous penetration testing. It combines offline RL on a curated dataset of 500+ real-world expert walkthroughs with online RL in interactive CTF environments, achieving state-of-the-art results on Cybench and AutoPenBench.
• Related Paper: Pentest-R1: Towards Autonomous Penetration Testing Reasoning Optimized via Two-Stage Reinforcement Learning
• GitHub: https://github.com/KHenryAegis/Pentest-R1
• Core Highlights: The project's key innovation is its two-stage synergy: offline GRPO training instills foundational attack logic from "Thought-Command-Observation" tuples, while online GRPO refines error correction through live environment interaction. The ablation study proves that both stages are essential for peak performance. Remarkably, fine-tuned from just an 8B model, Pentest-R1 rivals and even surpasses proprietary giants like GPT-4o and Claude, proving that strategic RL can outperform brute-force scaling.

---

💻 EnIGMA+

• Brief Introduction: EnIGMA+ is an enhanced evaluation scaffolding designed for cybersecurity agents to tackle Capture The Flag (CTF) challenges. Built upon the SWE-agent framework, it serves as the core agent execution component of the Cyber-Zero pipeline, enabling systematic assessment of how large language models handle complex, multi-step offensive security tasks.
• Related Paper: Cyber-Zero: Training Cybersecurity Agents Without Runtime
• GitHub: https://github.com/amazon-science/Cyber-Zero/tree/main/enigma-plus
• Core Highlights: The project introduces a runtime-free trajectory synthesis approach to democratize cybersecurity agent training. Key features include a centralized YAML-based configuration for seamless integration of diverse LLMs and a standardized evaluation protocol that prioritizes turn-based efficiency over cost. By providing repaired benchmark suites like InterCode-CTF and Cybench, EnIGMA+ ensures a stable and fair environment for measuring agent capabilities in capturing flags and generating accurate exploit trajectories.

---

💻 EnIGMA

• Brief Introduction: EnIGMA (Enhanced Interactive Generative Model Agent) is a specialized language model (LM) agent built upon the SWE-agent framework, designed to autonomously solve Capture The Flag (CTF) challenges and tackle offensive cybersecurity tasks.
• Related Paper: EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities
• GitHub: https://github.com/SWE-agent/SWE-agent
• Core Highlights: The framework introducing Interactive Agent Tools (IATs). These novel interfaces allow the agent to run and multitask with interactive command-line utilities—such as debuggers (gdb) and server connections—which previous agents could not handle. Additionally, it integrates a new Summarizer to manage long context windows and formally identifies a hallucination phenomenon termed "soliloquizing" (where the model generates its own fake observations), providing methods to quantify and mitigate it.

---

💻 Strix

• Brief Introduction: A fully custom-built multi-agent penetration testing framework that utilizes a tree-based agent architecture.
• GitHub: https://github.com/usestrix/strix
• Core Highlights: Features a Root Agent for orchestration and Sub-Agents for independent validation based on vulnerability types and components. It supports parallel execution of the discovery-validation-reporting-remediation chain. Incorporates 40+ built-in vulnerability skill files, uses Jinja2 to embed system prompts for knowledge injection, executes within a Docker Kali Linux sandbox for isolation, provides a terminal TUI visualization based on Textual, and supports multiple model backends via LiteLLM.

---

💻 PentAGI

• Brief Introduction: A full-stack autonomous penetration testing platform with a Go backend and React frontend, containing 14 specialized agents including Researcher, Developer, and Executor.
• GitHub: https://github.com/vxcontrol/PentAGI
• Core Highlights: Each agent is equipped with a 2-20KB independent system prompt template. It integrates 25+ tools (Docker isolated terminal, browser, 7 search engines), features pgvector vector storage and a Neo4j knowledge graph, and includes a complete observability stack (Grafana + VictoriaMetrics + Jaeger + Langfuse). It is built on a custom fork of langchaingo.

---

💻 CAI (Cybersecurity AI)

• Brief Introduction: A custom multi-agent framework comprising 20+ specialized red and blue team agents, supporting Parallel, Swarm, Hierarchical, Sequential, and Conditional collaboration modes.
• GitHub: https://github.com/aliasrobotics/cai
• Core Highlights: Enables context transfer between agents via a Handoff mechanism. Tools are organized by Kill Chain stages and support the MCP protocol for external tool integration (e.g., Burp Suite). It features multi-layer security protections (Input/Output Guardrails, Unicode homograph detection) and supports 300+ models via LiteLLM.

---

💻 Pentest-Swarm-AI

• Brief Introduction: A penetration testing system based on a Stigmergic Swarm architecture, utilizing a Blackboard shared state and Pheromone-decay indirect coordination mechanism without a central planner.
• GitHub: https://github.com/Armur-Ai/Pentest-Swarm-AI
• Core Highlights: Contains five autonomous responsive agents (Recon/Classifier/Exploit/Report/Seed). It supports both sequential pipeline and decentralized Swarm modes. Integrates 15+ security tools and features a comprehensive evidence capture and reporting system (screenshots, HTTP files, bounty estimation, multi-platform submission templates).

---

💻 pentest-copilot

• Brief Introduction: A fully custom three-tier cascaded agent framework designed for automated pentesting (Main Agent 25 rounds → Subagent 15 rounds/10 mins → Swarm Agent 25 rounds/15 mins).
• GitHub: https://github.com/bugbasesecurity/pentest-copilot
• Core Highlights: Includes 16 core tools, 5 Burp integration tools, and 5 orchestration tools covering bash execution, Python scripts, browser automation, and full Burp integration. Supports multi-model competition mechanisms (Swarm/Racer), structured EngagementState for decoupled conversational memory, and an on-demand installation registry for 100+ security capabilities.

---

💻 Apex

• Brief Introduction: An AI penetration testing tool built on Vercel AI SDK v6, utilizing a three-layer agent architecture (Harness base class + Specialized Agents + Orchestration tools).
• GitHub: https://github.com/pensarai/apex
• Core Highlights: Integrates 35+ tools covering the entire lifecycle. Supports Swarm parallel execution (up to 10 concurrency) and a deterministic three-stage workflow (Attack Surface Discovery → Swarm Penetration → Reporting) with resume capabilities. Features a cross-session persistent memory system, three-layer context compression, and dual interaction modes (OpenTUI + React and Headless CLI).

---

💻 AIDA

• Brief Introduction: A platform that transforms LLMs into autonomous pentesting agents via the MCP protocol, executing the complete attack chain within a Docker container.
• GitHub: https://github.com/Vasco0x4/AIDA
• Core Highlights: Provides 18 MCP tools and supports a three-level command approval mode (Auto/Semi-auto/Manual). Uses WebSockets for real-time execution result pushes. Features a model-agnostic architecture, automatic credential placeholder replacement, a React Web frontend, and an integrated AI Chat panel. Execution environments include built-in aida-pentest or Exegol.

---

💻 HackingBuddyGPT

• Brief Introduction: A minimalist LLM automated pentesting tool developed by TU Wien's academic lab, designed around the concept of implementing automated pentesting in just 50 lines of Python code.
• GitHub: https://github.com/IPA-Lab/HackingBuddyGPT
• Core Highlights: Features a pure Python custom architecture (UseCase → AutonomousUseCase → Agent → Capability) with 10 Agents covering Linux/Windows privilege escalation, Web pentesting, and REST API testing. Utilizes a simple ReAct loop, boasts lightweight dependencies, and remains completely model-agnostic.

---

💻 pentest-ai

• Brief Introduction: An end-to-end penetration testing CLI tool (available as PyPI package ptai) composed of 18 specialized agents.
• GitHub: https://github.com/0xSteph/pentest-ai
• Core Highlights: Uses a heuristic rule engine to automatically route target types (Web/AD/Cloud/Mobile). Wraps 200+ security tools and includes a built-in Playwright browser agent for SPA/JS-rendered pages. Automatically discovers and constructs vulnerability attack chains. Supports HITL REPL interaction, 6 report output formats, and native CI/CD integration.

---

💻 BugTraceAI

• Brief Introduction: A privacy-friendly autonomous security scanning framework containing 28 agents, with 14 expert exploitation agents covering vulnerabilities like XSS, SSRF, IDOR, and LFI.
• GitHub: https://github.com/BugTraceAI/BugTraceAI
• Core Highlights: Features a progressively upgrading detection model (e.g., XSS goes from single request to CDP validation across 6 levels), multi-role consensus voting for false positive filtering (rate <5%), and semantic deduplication based on LanceDB. Equipped with a React 18 Web console and WebSocket real-time event streaming.

---

💻 Xalgorix

• Brief Introduction: A zero-dependency, fully custom Go-based AI penetration agent deployed as a single binary, integrating 85+ security tools.
• GitHub: https://github.com/xalgord/xalgorix
• Core Highlights: Adopts a 22-stage methodology encoded in System Prompts, using a Hook system to enforce workflow (completion gating, stuck detection, active Skill Suggester). Features dual-layer page perception (semantic tree @eNN + advanced UI control discovery @xpaNN) and 750+ embedded SKILL.md knowledge files. Provides a full Web UI, Bubbletea TUI, and PDF report generation.

---

💻 DRAKBEN

• Brief Introduction: A 100% custom-built AI penetration testing framework featuring a unique three-layer cognitive memory system (Working Memory, Evolutionary Memory, and Cross-session Knowledge Base).
• GitHub: https://github.com/ahmetdrak/drakben
• Core Highlights: Incorporates a five-layer reflection system (ReAct loop, confidence scoring, cognitive introspection, SelfRefiningEngine strategy evolution, and hallucination detection). Contains 34 registered tools and uses SQLite + ChromaDB with a 90-day TTL. Equipped with a prompt_toolkit-based bilingual terminal REPL and FastAPI REST API.

---

💻 AutoPentest

• Brief Introduction: A LangGraph-based dual-layer reflection black-box penetration framework utilizing a Planner-Replanner-Supervisor architecture.
• GitHub: https://github.com/JuliusHenke/autopentest
• Core Highlights: Orchestrates 8 OWASP Top 10 specialized agents via StateGraph. Forces replanning after each execution and dynamically adjusts strategies based on feedback. Equipped with tools like PersistentShell, Playwright browser, and NVD API. Offers optional Pinecone vector database RAG support and uses LangSmith for experiment tracking.

---

💻 watchtower

• Brief Introduction: A three-agent penetration testing CLI tool based on LangGraph, LangChain, and LiteLLM, where Planner, Worker, and Analyst roles collaborate via a fixed-loop topology.
• GitHub: https://github.com/fzn0x/watchtower
• Core Highlights: Integrates 23 security tools invoked as subprocesses. Uses SQLite for persistent state management, natively supporting scan resumption. Features a built-in interactive tool selector, multi-LLM backend compatibility (OpenAI/Anthropic/Gemini/OpenRouter), and offline PDF report exportation capabilities.

---

💻 Autonomous Penetration Testing Copilot

• Brief Introduction: A single-file, purely hand-crafted LLM-driven penetration testing agent consisting of approximately 5,000 lines of Python code.
• GitHub: https://github.com/Krishcalin/Autonomous-Pen-Testing
• Core Highlights: Includes 30 tools covering the full lifecycle and supports both Claude tool_use and OpenAI function calling protocols. Features a six-stage discovery and validation pipeline and a cross-tool discovery correlation engine. Supports SSH remote or local invocation, includes a built-in stealth mode, breakpoint recovery, and utilizes pure CLI interaction.

---

💻 VulnClaw

• Brief Introduction: A completely self-developed AI penetration testing CLI tool (Python). The AgentCore class orchestrates LLM calls and tool execution, supporting two modes: single-shot penetration (15 rounds) and continuous penetration (100 rounds/cycle x 10 cycles).
• GitHub: https://github.com/Unclecheng-li/VulnClaw
• Core Highlights: 20 penetration Skills (7 core + 13 specialized) covering Web/Intranet/Android/CTF/OSINT scenarios, including 138 reference documents. 4 built-in tools + 11 MCP server configurations. The prominent feature is its anti-hallucination mechanism—hypothesis verification reminders, failure history tracking, forced path switching, and a CTF flag verification state machine, effectively preventing LLM hallucinations and infinite loops.

---

💻 AWE

• Brief Introduction: A memory-augmented multi-agent Web penetration testing framework, covering 10 vulnerability types (XSS/SQLi/SSTI/XXE/Command Injection/LFI/SSRF/IDOR/Default Credentials/Information Disclosure).
• Related Paper: AWE: Adaptive Agents for Dynamic Web Penetration Testing
• GitHub: https://github.com/stuxlabs/AWE
• Core Highlights: A three-tier orchestration architecture (IntelligentOrchestrator / EnhancedXSSOrchestrator / ConversationalAgent), with each vulnerability subsystem implementing a three-level escalation strategy (Rule Engine → Context Analysis → LLM Adaptive). Dual-layer memory system (short-term session + SQLite persistence), complete reasoning tracking (five-step decision loop). Achieved an 87% XSS success rate and a 66.7% blind SQLi success rate on the XBOW benchmark, reducing token consumption by 98% compared to its predecessor.

---

💻 CortexAI

• Brief Introduction: A Node.js-based autonomous penetration testing AI agent driven by an Azure OpenAI (GPT-4o) ReAct reasoning loop.
• GitHub: https://github.com/theelderemo/cortexai
• Core Highlights: ToolRegistry + PluginLoader registry pattern achieves complete decoupling of 26+ tools, invoked via OpenAI Function Calling format. Dual-layer page perception featuring Puppeteer dynamic rendering + static HTML regex fallback, supporting SPA route detection. Enterprise-grade project management (SQLite persistence, vulnerability lifecycle New→Confirmed→Remediated, HTTP evidence chain, OWASP/CWE classification). Complete audit chain via AgentLogger.

---

💻 Rogue

• Brief Introduction: An LLM penetration testing agent integrated with RAG knowledge enhancement, connecting to four authoritative security knowledge sources: PentestMonkey, CAPEC, OWASP WSTG, and CISA KEV, dynamically filtering relevant CVEs based on the scan context.
• GitHub: https://github.com/faizann24/rogue
• Core Highlights: Dual-channel network monitoring with Playwright browser automation + CDP protocol, automatically identifying tech stacks/JS libraries/CMS/API endpoints. Batch iterative planning mechanism, passing execution insights from previous batches to subsequent ones to achieve cross-batch adaptation. An independent LLM acts as a strict vulnerability evaluator, filtering false positives using HackerOne bounty standards. Built-in OWASP 2021 Top 10 baseline check plan.

---

💻 AI_Pentest

• Brief Introduction: A five-agent collaborative architecture (CoordinatorAgent + ReconAgent + VulnAgent + ExploitAgent + ReportAgent), with a Python/FastAPI backend and a comprehensive Web Dashboard frontend built with Vue 3 + TDesign.
• GitHub: https://github.com/yuanweipeifang/AI_Pentest
• Core Highlights: Integration of 30+ Kali tools. ExploitAgent implements an 800+ line automated Web attack chain (SQL Injection/XSS/Command Injection/Path Traversal/File Upload/Deserialization/XXE/SSRF), supporting multi-step chained exploitation. CoordinatorAgent multi-round collaborative planning + short-circuit convergence mechanism. Rich CLI interactive mode + WebSocket real-time push + multi-granularity progress callbacks.

---

💻 PentestAI-Google-ADK

• Brief Introduction: A penetration testing framework based on Google ADK v1.19.0, adopting a three-tier orchestration of Orchestrator + 2 composite Agents (Reconnaissance sequential execution / Vulnerability parallel scanning), totaling 7 sub-agents.
• GitHub: https://github.com/manishmitra017/Pentest-google-adk-agent
• Core Highlights: Strictly follows the PTES methodology, with a built-in comprehensive authorization verification system (IP ranges/Domain wildcards/CIDR), and enforces human_approved authorization for exploitation and privilege escalation operations. Strong security design awareness (legal warnings, DoS prohibition, exclusion of production environments). The project is in an early prototype stage; only 4 out of 11 tool functions have real implementations, while the rest are placeholders.

---

💻 Transilience-AI

• Brief Introduction: A skill-driven penetration testing framework based on Claude Code's native capabilities, featuring 26 skills covering OWASP Top 10 (100%), OWASP LLM Top 10 (100%), SANS Top 25 CWE (90+%), and 53 attack types.
• GitHub: https://github.com/transilienceai/communitytools
• Core Highlights: Three-role separation (Coordinator/Executor/Validator), with the Validator performing blind reviews to prevent confirmation bias. Three-layer file system memory (attack-chain.md / experiments.md / context-injection.md), 3-strike no-progress detection + P4b Reset mechanism. Creative research synthesized from three sources (model knowledge + skill cross-referencing + online research), with a symlink architecture to achieve skill reuse.

---

💻 CHeaT

• Brief Introduction: A defensive tool (not a pentesting agent), designed to defend networks against autonomous LLM-driven penetration testing agents by embedding adversarial payloads into network assets.
• GitHub: https://github.com/Daniel-Ayz/CHeaT
• Related Paper: Cloak, Honey, Trap: Proactive Defenses Against LLM Agents
• Core Highlights: Implements 6 defense strategies with 15 payload generation techniques (honeytokens + prompt injection). Tool wrapper mechanism hijacks system tool outputs to inject deceptive payloads when LLM agents execute commands. Zero external dependencies — pure Python standard library. Modular architecture (DefenseDatabase / DefenseCreator / DefenseInstaller). Covers strategies from "no vulnerabilities found" deception to honeypot credentials and counter-attack traps. Evaluated against PentestGPT with 11 CTF virtual machines.

---

💻 Cairn

• Brief Introduction: A general-purpose AI state-space search and problem-solving engine, validated first on autonomous penetration testing.
• GitHub: https://github.com/oritera/Cairn
• Core Highlights: Built on a Blackboard Architecture utilizing a dynamic fact-intent graph with three primitives (Fact, Intent, Hint). Agent Workers run a unified OODA loop (Observe, Orient, Decide, Act) to dynamically generate and execute tasks without predefined roles or workflows. Coordinates exclusively through shared-board stigmergy, eliminating direct communication and information silos. Achieved 3rd place and was the only team to AK (All Kill, 54/54) the 2nd Tencent Cloud Hackathon AI Penetration Testing Challenge. Zero MCP tools, zero RAG, and zero predefined agent roles. Supports Claude Code, Codex, and Pi backends under an AGPLv3 license.

---

ℹ️ 一键返回目录

MCP Tools

A collection of existing Model Context Protocol (MCP) servers designed to integrate penetration testing tools with LLM agents. Each integration typically requires the Core Tool and an MCP Server.

Reconnaissance

• 🛠️ Nmap: Core Tool (nmap.org) | MCP Server (gc-nmap-mcp)
  • Network discovery, port scanning, and security auditing.
• 🛠️ Amass: Core Tool (GitHub) | MCP Server (gc-amass-mcp)
  • In-depth network mapping and asset discovery.
• 🛠️ Assetfinder: Core Tool (GitHub) | MCP Server (gc-assetfinder-mcp)
  • Passive subdomain discovery tool.
• 🛠️ AlterX: Core Tool (GitHub) | MCP Server (gc-alterx-mcp)
  • Subdomain wordlist generator using DSL.
• 🛠️ Certificate Transparency: Core Service (crt.sh) | MCP Server (gc-crtsh-mcp)
  • Subdomain discovery using public certificate logs (API-based, no local tool required).
• 🛠️ Wayback URLs: Core Tool (GitHub) | MCP Server (gc-waybackurls-mcp)
  • Fetches URLs from the Wayback Machine.
• 🛠️ Gowitness: Core Tool (GitHub) | MCP Server (gc-gowitness-mcp)
  • Web screenshot and reconnaissance tool for capturing and analyzing web pages.

Scanning & Fuzzing

• 🛠️ httpx: Core Tool (GitHub) | MCP Server (gc-httpx-mcp)
  • Fast, multi-purpose HTTP/HTTPS toolkit.
• 🛠️ Nuclei: Core Tool (GitHub) | MCP Server (gc-nuclei-mcp)
  • Template-based vulnerability scanner.
• 🛠️ FFUF: Core Tool (GitHub) | MCP Server (gc-ffuf-mcp)
  • Fast web fuzzer for directory/file/parameter discovery.
• 🛠️ Katana: Core Tool (GitHub) | MCP Server (gc-katana-mcp)
  • Next-generation web crawler.
• 🛠️ SQLMap: Core Tool (sqlmap.org) | MCP Server (gc-sqlmap-mcp)
  • Automatic SQL injection and database takeover tool.
• 🛠️ Arjun: Core Tool (GitHub) | MCP Server (gc-arjun-mcp)
  • HTTP parameter discovery suite.
• 🛠️ Masscan: Core Tool (GitHub) | MCP Server (gc-masscan-mcp)
  • High-speed TCP port scanner.
• 🛠️ SSL Scan: Core Tool (GitHub) | MCP Server (gc-sslscan-mcp)
  • Scans SSL/TLS ciphers and configurations.
• 🛠️ HTTP Headers Security: Core Tool (owasp.org) | MCP Server (gc-http-headers-security-mcp)
  • Analyzer for HTTP security headers against OWASP standards.
• 🛠️ Smuggler: Core Tool (Github) | MCP Server (gc-smuggler-mcp)
  • Advanced tool for detecting HTTP Request Smuggling vulnerabilities.
• 🛠️ WPScan: Core Tool (Github) | MCP Server (gc-wpscan-mcp)
  • WordPress vulnerability scanner for detecting plugins, themes, and configuration issues.

Exploitation & Post-Exploitation

• 🛠️ Hydra: Core Tool (GitHub) | MCP Server (gc-hydra-mcp)
  • Network logon cracker (brute-force tool).
• 🛠️ Metasploit: Core Tool (metasploit.com) | MCP Server (gc-metasploit)
  • The world's most used penetration testing framework (via msfrpcd API).
• 🛠️ shuffledns: Core Tool (Github) | MCP Server (gc-suffledns-mcp)
  • High-speed and customizable DNS brute-forcing and resolution tool.

Comprehensive tool

• 🛠️ HexStrike: Core Tool (Github) | MCP Server (gc-hexstrike-mcp) | containerized distribution (hexstrike-ai-kit)
  • Multi-functional penetration testing agent.
• 🛠️ Kali: Core Tool (kali.org) | MCP Server (gc-kali-mcp)
  • A Linux virtual machine integrating multiple penetration tools.

ℹ️ Back to Top

Benchmarks

Datasets or benchmarks used to evaluate the performance of LLMs on penetration testing tasks.

• 📊 [XBOW] - Link

  • Brief intro: A professional benchmark set featuring 104 standalone CTF challenges, specifically designed to evaluate and test the capabilities of AI hacking agents or automated penetration testing tools.

• 📊 [CyberBattleSim] - Link

  • Brief intro: An open-source platform developed by Microsoft that utilizes a simulated network environment to experiment with and research the interactive behaviors of automated AI agents in cybersecurity attack and defense scenarios.

• 📊 [VulHub] - Link

  • Brief intro: Vulhub is an open-source collection of pre-built, ready-to-use vulnerable Docker environments. With just one command, you can launch a vulnerable environment for security research, learning, or demonstration, no prior Docker experience required.

• 📊 [DVWA] - Link

  • Brief intro: A PHP/MariaDB web application that is intentionally designed to be full of vulnerabilities. It provides a legal practice environment for testing skills, understanding, and learning common web application vulnerabilities.

• 📊 [VulnStack] - Link

  • Brief intro: A practical red team evaluation environment that includes several typical intranet penetration scenarios (such as domain penetration and vertical migration). Through pre-configured virtual machine images, it helps researchers evaluate the automated penetration and decision-making capabilities of AI agents in complex network topologies.

• 📊 [Pikachu] - Link

  • Brief intro: A web security testing environment with a vulnerability practice platform, covering a variety of common web vulnerabilities (such as injection, XSS, CSRF, file upload, etc.). Compared to DVWA, it has a more granular and diverse range of vulnerability types, making it ideal for evaluating LLM's ability to construct and understand various web attack payloads.

• 📊 [OWASP Benchmark] - Link

  • Brief intro: An open-source Java benchmark library for evaluating the accuracy of security scanning tools. It contains thousands of real and fake vulnerability test cases and can scientifically measure the performance and false positive rate of LLM in assisted code auditing (SAST/DAST) tasks by calculating recall and precision.

• 📊 [Hack The Box] - Link

  • Brief intro: A world-leading online cybersecurity training platform that provides a massive collection of live, gamified machines and challenges. It is widely used to evaluate the end-to-end penetration testing capabilities of AI agents in highly realistic, complex, and constantly updated environments, covering everything from initial foothold to privilege escalation.

• 📊 [NYU CTF Bench] - Link

  • Brief intro: A specialized benchmark designed by NYU researchers to evaluate LLMs in Capture The Flag (CTF) scenarios containing 200 challenges across 6 CTF categories. The CTF challenges are dockerized and easily deployable to allow an LLM-based automation framework to interact with the challenge and attempt a solution.

• 📊 [AUTOPENBENCH] - Link

  • Brief intro: An open-source benchmark for evaluating generative agents in automated penetration testing, featuring 33 Dockerized vulnerable tasks (in-vitro fundamentals and real-world CVEs from 2014-2024). It supports unrestricted command execution, provides standardized evaluation metrics (Success Rate/Progress Rate), and enables fair comparison of AI agents' penetration capabilities.

• 📊 [Cybench] - Link

  • Brief intro: An open-source benchmark for evaluating LLM cybersecurity capabilities and risks, featuring 40 professional CTF tasks (2022-2024 competitions) with subtask guidance and Kali Linux environment, supporting standardized performance comparison.

• 📊 [HackWorld] - Link

  • Brief intro: An open-source benchmark environment designed to evaluate the capabilities of computer-use agents in exploiting web application vulnerabilities, supporting VMware virtualization environments (Kali Linux images) and providing standardized evaluation workflows with configurable LLM/API key integration, as well as batch experiment execution support for AI hacking agents.

• 📊 [GOAD] - Link

  • Brief intro: A free and open-source pentest-focused Active Directory (AD) lab project maintained by Orange Cyberdefense, offering multiple pre-built vulnerable AD environments (GOAD Full/GOAD-Light/GOAD-Mini/SCCM/NHA/MINILAB) with diverse topologies (multi-forest/domain, varying VM counts). It preconfigures hundreds of AD-specific vulnerabilities and misconfigurations (e.g., Kerberos attacks, ACL abuse, ADCS flaws, NTLM relay, MSSQL trusted links) to evaluate AI agents' ability to exploit complex AD attack paths, from initial foothold to full domain compromise in realistic enterprise intranet scenarios.

• 📊 [PACEbench] - Link

  • Brief intro: The core research framework of this report emphasizes the difficulty of real-world vulnerabilities, environmental complexity, and defense strategies. It includes 32 real-world CVE challenges, covering four practical scenarios: single-point exploitation, cross-host chain attacks, and defense evasion.

• 📊 [CyberGym] - Link

  • Brief intro: CyberGym is a large-scale and realistic benchmark featuring 1,507 real-world vulnerabilities across 188 open-source projects. It primarily evaluates the ability of AI agents to generate proof-of-concept (PoC) tests to reproduce vulnerabilities, given only the vulnerability's text description and the corresponding codebase. The benchmark is designed with four difficulty levels, covering various evaluation scenarios from open-ended vulnerability discovery without prior knowledge, to simulating 1-day vulnerability exploitation by combining stack traces or patch information. Furthermore, CyberGym is not only used for static capability assessment but has also successfully led to the discovery of 34 zero-day vulnerabilities and 18 historically incomplete patches, producing direct real-world security impact.

  ℹ️ Back to Top

Citation

Please cite our paper if you find this repository or our paper useful in your work:

@misc{peng2026hackershallucinatorscomprehensiveanalysis,
      title={Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing}, 
      author={Jiaren Peng and Zeqin Li and Chang You and Yan Wang and Hanlin Sun and Xuan Tian and Shuqiao Zhang and Junyi Liu and Jianguo Zhao and Renyang Liu and Haoran Ou and Yuqiang Sun and Jiancheng Zhang and Yutong Jiao and Kunshu Song and Chao Zhang and Fan Shi and Hongda Sun and Rui Yan and Cheng Huang},
      year={2026},
      eprint={2604.05719},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2604.05719}, 
}

Follow Us