Add Lambda-based compilation endpoint system #1676
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements Lambda functions to handle /api/compilers/<compilerid>/compile and /api/compilers/<compilerid>/cmake endpoints via SQS queues and WebSocket responses. Features: - Python Lambda functions for beta/staging/prod environments - SQS FIFO queues for reliable message handling - ALB integration with target groups and listener rules - Support for both JSON and plain text requests/responses - Content negotiation based on Accept headers - Comprehensive test coverage with 23 unit tests - Terraform infrastructure split into logical modules - Makefile integration for build, test, and deployment 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…umentation Features: - Beta environment auto-scaling based on compilation queue depth (3 msg/instance target) - Blue-green ASG support with shared queue monitoring - Comprehensive Lambda compilation workflow documentation - Code formatting and linting fixes for Lambda function 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Remove hardcoded environment constraints and path pattern logic: - Remove environment validation restricting to beta/staging/prod - Add alb_path_patterns variable for flexible path configuration - Replace hardcoded prod vs non-prod path logic with user-defined patterns - Update all environment configurations to specify explicit path patterns - Update documentation and examples to reflect generic approach Benefits: - Module can be used with any environment name (dev, test, local, etc.) - Supports custom path patterns and URL schemes - Truly reusable across different projects and use cases - Follows Terraform module best practices for genericity 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Correct WebSocket URLs to match existing infrastructure pattern: - Beta: wss://events.compiler-explorer.com/beta - Staging: wss://events.compiler-explorer.com/staging - Production: wss://events.compiler-explorer.com/prod This aligns with the AArch64 execution workflow which uses the same events.compiler-explorer.com WebSocket infrastructure for receiving asynchronous execution results. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Set visibility timeout to 5 seconds (down from 2 minutes) - Set message retention to 60 seconds (down from 30 minutes) These shorter timeouts are more appropriate for the Lambda-based compilation system where fast message processing and quick cleanup of unprocessed messages is desired. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements a Lambda-based compilation endpoint system to handle
/api/compilers/<compilerid>/compileand/api/compilers/<compilerid>/cmakeendpoints. This is part of the infrastructure work to support the compilation queue system described in compiler-explorer/compiler-explorer#7864.Key Features
• Lambda Functions: Python 3.12 Lambda functions for beta, staging, and prod environments
• SQS Integration: FIFO queues for reliable message handling with 30-minute retention
• WebSocket Response Handling: Async response retrieval from events system with configurable retry logic
• Content Negotiation: Support for both JSON and plain text requests/responses based on Accept headers
• ALB Integration: Target groups and listener rules for routing (beta active, staging/prod prepared)
• Comprehensive Testing: 23 unit tests covering all scenarios including error cases
• Terraform Infrastructure: Modular configuration split into logical files (security, ALB, SQS, Lambda)
Architecture
Implementation Details
• Timeout: 90 seconds for WebSocket response, 120 seconds Lambda timeout
• Retry Logic: Configurable retry count (default: 2) for WebSocket connections
• Request Handling: Preserves original headers and supports both JSON/plain text bodies
• Response Formatting: JSON response by default, plain text assembly when Accept: text/plain
• Environment Configuration: Separate queues and WebSocket URLs per environment
Testing
All functionality is covered by unit tests including:
Deployment
• Beta Environment: Ready for immediate deployment with active listener rules
• Staging/Production: Infrastructure ready, listener rules commented out for gradual rollout
• Makefile Integration: Build, test, and upload targets integrated with existing workflow
Related to: compiler-explorer/compiler-explorer#7864
Test plan
/beta/api/compilers/gcc/compileendpoint🤖 Generated with Claude Code