Architecture Overview

System Overview

QA Studio is a serverless web application for AI-powered automated testing built on Amazon Nova Act. The system consists of a React frontend, serverless API backend, ECS-based test execution workers, and a CLI tool. The qa-studio CLI is the single execution runtime: developer-run tests, CI-run tests, and cloud-triggered ECS worker tests all invoke the same code path.

Key Capabilities:

Natural language test creation and management
AI-powered browser automation with Amazon Nova Act
Mobile app testing on real devices via AWS Device Farm
Interactive test wizard with live browser preview
Test suite organization and execution
Comprehensive artifact capture (videos, screenshots, logs, traces)
OAuth 2.0 authentication for users and API clients
CLI tool for local test execution and management, and as the runtime for the cloud worker

System Components

graph TB
    subgraph "Client Layer"
        WebUI[React Web App<br/>CloudFront + S3]
        CLI[QA Studio CLI<br/>Python Tool]
    end
    
    subgraph "AWS Cloud"
        subgraph "Authentication"
            Cognito[AWS Cognito<br/>User Pool]
        end
        
        subgraph "API Layer"
            APIGW[API Gateway<br/>REST API]
            Authorizer[Lambda<br/>Authorizer]
        end
        
        subgraph "Business Logic"
            CreateUsecase[Lambda<br/>create_usecase]
            ExecUsecase[Lambda<br/>execute_usecase]
            ExecSuite[Lambda<br/>execute_test_suite]
            UpdateStatus[Lambda<br/>update_execution_status]
            GenArtifact[Lambda<br/>generate_artifact_url]
            OAuthMgmt[Lambda<br/>OAuth Management]
        end
        
        subgraph "Test Execution"
            SQS[SQS Queue<br/>Execution Queue]
            ECS[ECS Fargate<br/>Worker Tasks]
            CLIRuntime["qa-studio CLI<br/>(shared runtime)"]
            NovaAct[Nova Act SDK<br/>+ Playwright]
            AgentCore[Bedrock AgentCore<br/>Browser Tool]
            DeviceFarm[AWS Device Farm<br/>Mobile Devices]
            RecordingQueue[SQS Queue<br/>Recording Downloads]
            RecordingLambda[Lambda<br/>download_recording]
        end
        
        subgraph "Data Layer"
            DynamoDB[(DynamoDB<br/>Single Table)]
            S3[(S3 Bucket<br/>Artifacts)]
        end
    end
    
    WebUI -->|OAuth PKCE| Cognito
    CLI -->|OAuth PKCE| Cognito
    CLIRuntime -->|OAuth M2M<br/>client_credentials| Cognito
    
    WebUI -->|API Requests| APIGW
    CLI -->|API Requests| APIGW
    CLIRuntime -->|API Requests| APIGW
    
    APIGW --> Authorizer
    Authorizer --> Cognito
    
    APIGW --> CreateUsecase
    APIGW --> ExecUsecase
    APIGW --> ExecSuite
    APIGW --> UpdateStatus
    APIGW --> GenArtifact
    APIGW --> OAuthMgmt
    
    CreateUsecase --> DynamoDB
    ExecUsecase --> DynamoDB
    ExecUsecase --> SQS
    ExecSuite --> DynamoDB
    ExecSuite --> SQS
    UpdateStatus --> DynamoDB
    GenArtifact --> S3
    OAuthMgmt --> Cognito
    
    SQS --> ECS
    ECS --> CLIRuntime
    CLIRuntime --> NovaAct
    CLIRuntime --> AgentCore
    NovaAct --> AgentCore
    CLIRuntime -->|Mobile tests| DeviceFarm
    CLIRuntime -->|Enqueue via API| RecordingQueue
    RecordingQueue --> RecordingLambda
    RecordingLambda -->|Download video| DeviceFarm
    RecordingLambda -->|Upload video| S3
    RecordingLambda --> DynamoDB
    
    CLI -->|Local Execution| NovaAct
    CLI -->|Mobile tests| DeviceFarm
    
    style WebUI fill:#e1f5ff
    style CLI fill:#e1f5ff
    style CLIRuntime fill:#e1f5ff
    style Cognito fill:#fff4e1
    style APIGW fill:#ffe1e1
    style ECS fill:#ffe1f5
    style DynamoDB fill:#e1ffe1
    style S3 fill:#e1ffe1

Note on the cloud worker: batch ECS tasks run qa-studio run as their entrypoint (via the CLI-unified-runner refactor). The container reads execution context from ECS env vars, authenticates against Cognito using an M2M client whose credentials live in Secrets Manager, and writes all state back through the public API — the worker no longer talks to DynamoDB, S3, SQS or EventBridge directly for batch executions. Wizard-mode tasks still use the legacy wizard_worker.py path until the wizard migration lands (separate spec).

Data Flow Diagrams

Web UI Authentication Flow

sequenceDiagram
    participant User
    participant WebUI as React Web App
    participant Cognito as AWS Cognito
    
    User->>WebUI: Access application
    WebUI->>Cognito: Redirect to Cognito Hosted UI
    User->>Cognito: Enter credentials
    Cognito->>WebUI: Redirect with authorization code
    WebUI->>Cognito: Exchange code for tokens (PKCE)
    Cognito->>WebUI: Access token + ID token
    WebUI->>WebUI: Store tokens in memory

CLI Authentication Flow

sequenceDiagram
    participant User
    participant CLI as QA Studio CLI
    participant Browser
    participant Cognito as AWS Cognito
    participant LocalServer as Local Callback Server
    
    User->>CLI: qa-studio login
    CLI->>LocalServer: Start local server (port 8080)
    CLI->>Browser: Open Cognito Hosted UI
    User->>Browser: Enter credentials
    Browser->>Cognito: Submit credentials
    Cognito->>LocalServer: Redirect with authorization code
    LocalServer->>CLI: Pass authorization code
    CLI->>Cognito: Exchange code for tokens (PKCE)
    Cognito->>CLI: Access token + refresh token
    CLI->>CLI: Store tokens in ~/.qa-studio/tokens.json

Test Execution Flow (Cloud)

sequenceDiagram
    participant User
    participant WebUI as Web UI / CLI
    participant API as API Gateway
    participant Lambda as Execute Lambda
    participant SQS as SQS Queue
    participant ECS as ECS Worker
    participant Entrypoint as entrypoint.sh
    participant RunnerCLI as qa-studio run
    participant AgentCore as Bedrock AgentCore Browser
    participant NovaAct as Nova Act SDK
    participant EventBridge as EventBridge
    
    User->>WebUI: Trigger test execution
    WebUI->>API: POST /usecase/{id}/execute
    API->>Lambda: Invoke
    Lambda->>API: Create execution record (via DynamoDB)
    Lambda->>SQS: Send execution message
    Lambda->>WebUI: Return execution ID
    
    SQS->>ECS: Trigger worker task
    ECS->>Entrypoint: Start container (env: USECASE_ID, EXECUTION_ID, …)
    Entrypoint->>Entrypoint: Read API URL + token endpoint from SSM
    Entrypoint->>RunnerCLI: exec qa-studio run --browser agentcore …
    
    RunnerCLI->>API: Cognito M2M client_credentials grant<br/>GET execution, steps, variables
    RunnerCLI->>AgentCore: Provision browser
    AgentCore->>RunnerCLI: CDP endpoint + live-view URL
    RunnerCLI->>API: POST live-view URL
    
    loop For each step
        RunnerCLI->>NovaAct: Execute step with nova.act()
        NovaAct->>AgentCore: Drive browser over CDP
        NovaAct->>AgentCore: Save action traces (S3Writer / API)
        RunnerCLI->>API: PATCH step status
        RunnerCLI->>API: Upload step artifact (presigned PUT)
        Note over RunnerCLI,API: For retrieve_value / transform:<br/>POST runtime-variable per-capture
    end
    
    RunnerCLI->>API: PATCH execution status (success/failed)
    API->>EventBridge: Emit usecase.execution.completed
    RunnerCLI->>API: DELETE live-view
    RunnerCLI->>AgentCore: Delete browser
    ECS->>ECS: Container exits with CLI's exit code
    
    Note over EventBridge: Event triggers downstream<br/>processes (e.g., cache building)

Test Execution Flow (Local CLI)

sequenceDiagram
    participant User
    participant CLI as QA Studio CLI
    participant API as API Gateway
    participant NovaAct as Nova Act SDK
    participant DynamoDB
    participant S3
    
    User->>CLI: qa-studio run --usecase-id test-123
    CLI->>API: GET /steps/{usecase_id}
    API->>DynamoDB: Fetch test steps
    DynamoDB->>API: Return steps
    API->>CLI: Return steps
    
    alt Local-only mode
        CLI->>CLI: Execute locally (no remote record)
    else Cloud mode
        CLI->>API: POST /usecase/{id}/execute?trigger-type=ci_runner
        API->>DynamoDB: Create execution record
        API->>CLI: Return execution ID
    end
    
    loop For each step
        CLI->>NovaAct: Execute step with nova.act()
        NovaAct->>NovaAct: AI determines actions
        NovaAct->>NovaAct: Execute browser actions
        CLI->>CLI: Save artifacts locally
        
        alt Cloud mode
            CLI->>API: POST /artifact/upload (get presigned URL)
            API->>S3: Generate presigned URL
            API->>CLI: Return presigned URL
            CLI->>S3: Upload artifact
            CLI->>API: PATCH /execution/{id}/step/{step_id}/status
            API->>DynamoDB: Update step status
        end
    end
    
    alt Cloud mode
        CLI->>API: PATCH /execution/{id}/status
        API->>DynamoDB: Update execution status
    end
    
    CLI->>User: Display results

Component Details

Frontend (React Web App)

Technology: React 18, TypeScript, Vite, AWS Cloudscape Design System

Hosting: CloudFront (CDN) + S3 (static hosting)

Key Features:

Test creation and management UI
Interactive test wizard with live browser preview
Test suite organization
Execution history and artifact viewing
OAuth client management
User management

Authentication: OAuth 2.0 PKCE flow via Cognito Hosted UI

CLI Tool (QA Studio CLI)

Technology: Python 3.11+, Click framework

Installation: pip install -e ./qa-studio-cli[runner]

Key Features:

Browser-based OAuth authentication
Test and suite management commands
Local test execution with Nova Act (web and mobile)
Mobile test execution via AWS Device Farm
Configuration management
Kiro IDE integration

Authentication: OAuth 2.0 PKCE flow with local callback server

API Layer

Technology: Amazon API Gateway (REST API) + AWS Lambda (Python 3.11)

Authentication:

Lambda authorizer validates JWT tokens from Cognito
Scope-based authorization (e.g., api/usecase.read, api/usecase.write)

Key Endpoints:

/usecases - Test management (CRUD)
/steps - Test step management
/suites - Test suite management
/executions - Execution management and triggering
/devices - Device Farm device listing
/oauth-clients - OAuth client management
/artifacts - Artifact URL generation

Test Execution (ECS Workers)

Technology: Amazon ECS with Fargate, Python 3.13, the qa-studio CLI (with [runner,agentcore] extras), Nova Act SDK, Playwright, Appium (mobile). Browser execution runs in Bedrock AgentCore, not inside the container.

Trigger: SQS messages from execute_usecase or execute_test_suite Lambdas.

Container contract:

The worker image installs the qa-studio CLI from the sibling qa-studio-cli/ package during Docker build.
entrypoint.sh runs on container start. It branches on WORKER_MODE:
- batch (default) — resolves the API URL + Cognito token endpoint from SSM, reads OAuth client-credentials from env (injected by ECS from Secrets Manager), and execs qa-studio run --browser agentcore ….
- wizard — defers to wizard_worker.py until the wizard migration lands.

Execution Flow (Web, batch mode):

Entrypoint resolves QA_STUDIO_API_URL (SSM) and OAUTH_TOKEN_ENDPOINT (SSM), and exports OAUTH_CLIENT_ID / OAUTH_CLIENT_SECRET from the ECS secrets: injection.
qa-studio run --browser agentcore --trigger-type worker starts.
The CLI authenticates via Cognito M2M client-credentials and fetches execution details over the public API.
The AgentCore browser provisioner calls Bedrock to create + start a remote browser, returns CDP endpoint + live-view URL.
The runner publishes the live-view URL via the live-view endpoint so the frontend can show it.
For each step, the runner executes via NovaAct, posts step status + uploads artifacts via presigned PUT URLs, and (for retrieve_value / transform steps) publishes captured runtime variables per-capture.
On success or failure, the runner PATCHes the final execution status — the server-side update_execution_status Lambda emits the usecase.execution.completed EventBridge event.
The AgentCore browser + live-view record are torn down in the runner's finally block.

Execution Flow (Mobile, batch mode):

Steps 1-3 as above.
Instead of provisioning an AgentCore browser, the CLI builds a DeviceFarmActuator and hands it to NovaAct (uploads the app binary to Device Farm if needed).
After the Device Farm session starts, the runner persists the session ARN via the mobile-metadata endpoint so the frontend can link to Device Farm.
Steps execute via the actuator; all state writes go through the API.
After the session stops, the runner posts the final session ARN and calls the download-recording endpoint which enqueues a delayed SQS message (5 min) for the recording download Lambda.

Recording Download (Mobile):

The download_device_farm_recording Lambda picks up the delayed SQS message, waits for the session to reach a terminal state, downloads the video, uploads it to S3, and creates a DynamoDB artifact record.
If the session hasn't finalized yet, the Lambda raises an error and SQS retries after the visibility timeout.

Execution Flow (Mobile):

Receive execution message from SQS
Fetch test steps from DynamoDB
Provision a Device Farm remote access session (uploads app binary if needed)
Connect to the device via Appium through the Device Farm endpoint
Initialize Nova Act with DeviceFarmActuator
Execute each step with nova.act(instruction)
Stop the Device Farm session and enqueue a delayed SQS message for recording download
Update execution status in DynamoDB
Emit EventBridge event for downstream processing

Recording Download (Mobile):

After the worker stops the Device Farm session, it sends a delayed SQS message (5 min) to the recording download queue
A Lambda (download_device_farm_recording) picks up the message after the delay
The Lambda waits for the session to reach a terminal state, downloads the video artifact from Device Farm, uploads it to S3, and creates a DynamoDB artifact record
If the session hasn't finalized yet, the Lambda raises an error and SQS retries after the visibility timeout (3 min), up to 10 retries

Event Emission:

usecase.execution.completed is emitted by the update_execution_status Lambda (Source=qa-studio.api) on every terminal status transition.
Event detail: usecase_id, execution_id, execution_status, timestamp.
Triggers downstream processes such as cache building for step optimization.

Artifacts Generated:

Screenshots (PNG)
Videos (MP4) — from AgentCore Browser (web) or Device Farm (mobile)
Action traces (JSON and HTML)
Execution logs (text)
Trajectory JSONs (web) — persisted per step by the runner so subsequent runs can replay them without hitting Nova Act's LLM.

Data Storage

DynamoDB (Single Table Design):

Primary Key: pk (partition key)
Sort Key: sk (sort key)
Access patterns optimized with GSIs

Key Record Types:

USECASE#{id} / METADATA - Test metadata
USECASE#{id} / STEP#{id} - Test steps
USECASE_EXECUTION#{usecase_id} / EXECUTION#{id} - Execution records
EXECUTION#{id} / EXECUTION_STEP#{id} - Execution step records
SUITE#{id} / METADATA - Suite metadata
SUITE#{id} / USECASE#{id} - Suite-usecase mappings

S3 Bucket:

Artifacts organized by execution ID
Presigned URLs for secure uploads
Lifecycle policies for cost optimization

Security Architecture

Authentication

Users (Web UI + CLI):

OAuth 2.0 Authorization Code flow with PKCE
Cognito Hosted UI for login
Access tokens (1 hour) + refresh tokens (30 days)

API Clients (M2M):

OAuth 2.0 Client Credentials flow
Client ID + Client Secret
Access tokens (1 hour, no refresh)

Authorization

Scope-Based Access Control:

api/usecase.read - Read tests
api/usecase.write - Create/update/delete tests
api/suite.read - Read suites
api/suite.write - Create/update/delete suites
api/execution.read - Read execution results
api/execution.write - Trigger executions
api/oauth.read - Read OAuth clients
api/oauth.write - Create/manage OAuth clients

Lambda Authorizer:

Validates JWT signature
Checks token expiration
Extracts scopes from token
Caches authorization decisions (5 minutes)

Data Protection

Encryption at Rest:

DynamoDB: AWS-managed encryption
S3: AES-256 encryption
Secrets Manager: AWS-managed encryption

Encryption in Transit:

TLS 1.2+ for all API communication
HTTPS-only CloudFront distribution

Secret Management:

Test secrets stored in AWS Secrets Manager
Referenced by secret key in test steps
Retrieved at execution time only

Infrastructure Compliance (cdk-nag)

All CDK stacks are validated at synth time using cdk-nag with the AwsSolutionsChecks rule pack. This runs automatically during every cdk synth and cdk deploy.

Rules are applied via Aspects.of(app).add(new AwsSolutionsChecks(...)) in bin/cdk.ts
Known acceptable patterns are suppressed with documented reasons in lib/cdk-nag-suppressions.ts
Legacy cfn_nag suppressions for CDK-managed resources live in lib/cfn-nag-suppressions.ts
Any new resource that violates an AwsSolutions rule will surface as an error annotation during synth

Scalability & Performance

API Layer

API Gateway: Auto-scales to handle request volume
Lambda: Concurrent execution limit (default 1000)
DynamoDB: On-demand capacity mode

Test Execution

ECS: Auto-scales based on SQS queue depth
Parallel execution: Multiple workers process tests simultaneously
SQS: Buffers execution requests during high load

Artifact Storage

S3: Unlimited storage capacity
CloudFront: Global CDN for fast artifact access
Presigned URLs: Direct uploads bypass API Gateway limits

Monitoring & Observability

CloudWatch Logs

API Gateway access logs
Lambda function logs
ECS task logs
Worker execution logs

CloudWatch Metrics

API request count and latency
Lambda invocation count and duration
ECS task count and CPU/memory usage
SQS queue depth and message age

X-Ray Tracing

End-to-end request tracing
Service map visualization
Performance bottleneck identification

Cost Optimization

Compute

Lambda: Pay per invocation (sub-second billing)
ECS Fargate: Pay per task execution time
API Gateway: Pay per request

Storage

S3: Lifecycle policies to transition old artifacts to Glacier
DynamoDB: On-demand pricing for variable workloads

Data Transfer

CloudFront: Reduced data transfer costs
S3 presigned URLs: Direct uploads avoid API Gateway costs

Disaster Recovery

Backup Strategy

DynamoDB: Point-in-time recovery enabled
S3: Versioning enabled for critical artifacts
CloudFormation: Infrastructure as code for rapid rebuild

High Availability

Multi-AZ deployment for all services
CloudFront: Global edge locations
API Gateway: Built-in redundancy

Future Enhancements

Test result analytics and trends
Slack/email notifications
Test result comparison
Performance testing support
Android emulator support for local mobile testing
Device Farm session live streaming

Navigation Menu

Uh oh!

Architecture Overview

System Overview

System Components

Data Flow Diagrams

Web UI Authentication Flow

CLI Authentication Flow

Test Execution Flow (Cloud)

Test Execution Flow (Local CLI)

Component Details

Frontend (React Web App)

CLI Tool (QA Studio CLI)

API Layer

Test Execution (ECS Workers)

Data Storage

Security Architecture

Authentication

Authorization

Data Protection

Infrastructure Compliance (cdk-nag)

Scalability & Performance

API Layer

Test Execution

Artifact Storage

Monitoring & Observability

CloudWatch Logs

CloudWatch Metrics

X-Ray Tracing

Cost Optimization

Compute

Storage

Data Transfer

Disaster Recovery

Backup Strategy

High Availability

Future Enhancements

Uh oh!

FilesExpand file tree

architecture.md

Latest commit

History

architecture.md

File metadata and controls

Architecture Overview

System Overview

System Components

Data Flow Diagrams

Web UI Authentication Flow

CLI Authentication Flow

Test Execution Flow (Cloud)

Test Execution Flow (Local CLI)

Component Details

Frontend (React Web App)

CLI Tool (QA Studio CLI)

API Layer

Test Execution (ECS Workers)

Data Storage

Security Architecture

Authentication

Authorization

Data Protection

Infrastructure Compliance (cdk-nag)

Scalability & Performance

API Layer

Test Execution

Artifact Storage

Monitoring & Observability

CloudWatch Logs

CloudWatch Metrics

X-Ray Tracing

Cost Optimization

Compute

Storage

Data Transfer

Disaster Recovery

Backup Strategy

High Availability

Future Enhancements