🤖 AI Incident Triage Agent

An agentic AI that triages a live Cloud Run incident on its own — checking service health, querying Cloud Logging for recent errors, identifying the dominant failure pattern, and writing a full root-cause-analysis report, all autonomously.

Cloud Advocate · GK

---

🧠 Architecture

YOUR LAPTOP
└── triage_agent.py
    ├── Claude claude-sonnet-4-6 (tool_use loop)
    ├── tool: get_service_status()   → checks Cloud Run revision
    ├── tool: query_recent_errors()  → reads Cloud Logging
    └── tool: check_error_pattern()  → identifies root cause
                    ↓
GOOGLE CLOUD
└── Cloud Run: payment-api (buggy service)
    └── Cloud Logging: error logs from /crash endpoint

---

📋 Prerequisites

• gcloud CLI installed
• GCP project with billing enabled
• Python 3.10+
• Anthropic API key

---

1️⃣ Clone the Repo

git clone https://github.com/cloudadvocate/incident-triage-agent
cd incident-triage-agent

---

2️⃣ Configure GCP

Authenticate and set your project:

gcloud auth login
gcloud auth application-default login
gcloud config set project YOUR_PROJECT_ID

Enable the APIs needed for this demo:

gcloud services enable run.googleapis.com
gcloud services enable logging.googleapis.com

Confirm your active project:

gcloud config get-value project

---

3️⃣ Deploy the Buggy Service

Review the service code before deploying:

cat buggy-service/main.py

Build and deploy to Cloud Run:

gcloud run deploy payment-api \
  --source ./buggy-service \
  --region us-central1 \
  --allow-unauthenticated

Once deployed, grab the service URL:

gcloud run services describe payment-api \
  --region us-central1 \
  --format "value(status.url)"

Save it as an environment variable:

export PAYMENT_API_URL=https://YOUR-URL-FROM-ABOVE

Verify the service is healthy:

curl $PAYMENT_API_URL/
# Expected: {"status": "ok", "service": "payment-api"}

---

4️⃣ Generate the Incident

Hit the crash endpoint 5 times to create real error logs:

for i in {1..5}; do
  curl $PAYMENT_API_URL/crash
  sleep 1
done

Verify errors landed in Cloud Logging — open the GCP console and run this filter:

resource.type="cloud_run_revision"
resource.labels.service_name="payment-api"
severity=ERROR

Wait ~30 seconds for log ingestion before running the agent.

---

5️⃣ Set Up the Triage Agent

Install dependencies:

cd triage-agent
pip install -r requirements.txt

Set environment variables:

export ANTHROPIC_API_KEY=your-anthropic-api-key
export GOOGLE_CLOUD_PROJECT=your-project-id
export CLOUD_RUN_REGION=us-central1

---

6️⃣ Run the Agent

python triage_agent.py --service payment-api

You will see each tool call as it happens:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  AI Incident Triage Agent
  Service : payment-api
  Model   : claude-sonnet-4-6
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

▶ Calling tool : get_service_status
  Input        : {"service_name": "payment-api"}
  Result       : revision=payment-api-00001-abc ready=True

▶ Calling tool : query_recent_errors
  Input        : {"service_name": "payment-api", "minutes": 10}
  Result       : 5 error(s) found

▶ Calling tool : check_error_pattern
  Input        : {"log_entries": [...]}
  Result       : dominant_error=DatabaseConnectionPoolExhausted

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  RCA REPORT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
...
✓ Report saved → rca_payment-api_20260622_143022.md

Open the generated report:

cat rca_payment-api_*.md

---

📁 Project Structure

incident-triage-agent/
├── README.md                  ← you are here
├── buggy-service/
│   ├── main.py                ← Flask app: / returns 200, /crash returns 500
│   ├── requirements.txt
│   └── Dockerfile
└── triage-agent/
    ├── triage_agent.py        ← Claude tool_use loop + markdown report writer
    ├── tools.py               ← get_service_status, query_recent_errors, check_error_pattern
    └── requirements.txt

---

🔍 Troubleshooting

Agent can't read logs — permission denied

Grant the logging viewer role to your account:

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="user:YOUR_EMAIL" \
  --role="roles/logging.viewer"

No errors found by the agent

Cloud Logging has a ~30 second ingestion delay. Wait a minute after hitting /crash before running the agent.

get_service_status fails

Make sure GOOGLE_CLOUD_PROJECT is set and gcloud is authenticated:

gcloud auth list
gcloud config get-value project

---

🧹 Cleanup

Delete the Cloud Run service when done:

gcloud run services delete payment-api \
  --region us-central1 \
  --quiet

Delete the container images built during gcloud run deploy --source (stored in the auto-created cloud-run-source-deploy Artifact Registry repo):

gcloud artifacts repositories delete cloud-run-source-deploy \
  --location us-central1 \
  --quiet

If you added the logging viewer IAM binding from the Troubleshooting section, remove it:

gcloud projects remove-iam-policy-binding YOUR_PROJECT_ID \
  --member="user:YOUR_EMAIL" \
  --role="roles/logging.viewer"

(Optional) Disable the APIs enabled for this demo, if not used elsewhere in the project:

gcloud services disable run.googleapis.com logging.googleapis.com --quiet

Remove local generated reports and unset environment variables:

rm -f rca_payment-api_*.md
unset PAYMENT_API_URL ANTHROPIC_API_KEY GOOGLE_CLOUD_PROJECT CLOUD_RUN_REGION

---

👨‍💻 About

Built by GK — Cloud Architect and DevSecOps practitioner with 10+ years of enterprise networking and telecom experience.

Cloud Advocate is a YouTube channel covering AI on Cloud, DevSecOps, Multi-cloud, FinOps, and Cloud Career guidance with real-world hands-on depth.

📺 YouTube — Cloud Advocate 💼 LinkedIn

---

If this helped you, give the repo a ⭐ and subscribe to Cloud Advocate for more hands-on cloud and DevOps content.