Cloud provider abstraction layer

[UJ2202]

User Story

As a developer, I want a cloud provider abstraction layer so that CMBCluster can seamlessly work with different cloud providers without duplicating logic or requiring extensive code changes.

Description

Create a unified abstraction layer that provides consistent interfaces for cloud-specific operations, enabling CMBCluster to work with GCP, AWS, and future cloud providers through a common API while hiding implementation details.

Current Implementation Analysis

Based on codebase analysis, cloud-specific operations include:

• Storage Management: GCS bucket operations in backend/storage_manager.py
• Authentication: Workload Identity configuration in deployment scripts
• Container Registry: Artifact Registry references in Helm charts
• Infrastructure: GCP-specific setup and deployment scripts

Abstraction Layer Requirements

Create interfaces and implementations for:

1. Storage Operations

   • Bucket creation, deletion, and management
   • File upload/download operations
   • Access control and permissions
   • Unified API for GCS and S3

2. Authentication Management

   • Service account creation and configuration
   • Workload Identity / IRSA setup
   • Permission and role management
   • Token and credential handling

3. Container Registry Operations

   • Repository creation and management
   • Image push/pull operations
   • Registry authentication
   • Cross-cloud image mirroring

4. Infrastructure Provisioning

   • Cluster setup and configuration
   • Networking and security setup
   • Resource monitoring and management
   • Provider-specific optimizations

Technical Design

# Example abstraction interfaces
class CloudProvider(ABC):
    @abstractmethod
    def create_storage_bucket(self, name: str, config: dict) -> StorageBucket
    @abstractmethod
    def setup_workload_identity(self, config: dict) -> ServiceAccount
    @abstractmethod
    def create_registry_repository(self, name: str) -> Registry

class GCPProvider(CloudProvider):
    # GCP-specific implementations using GCS, Workload Identity, Artifact Registry

class AWSProvider(CloudProvider):
    # AWS-specific implementations using S3, IRSA, ECR

Acceptance Criteria

• Cloud provider interface is defined with all required operations
• GCP provider implementation maintains existing functionality
• AWS provider implementation provides equivalent capabilities
• Configuration system supports provider-specific settings
• Unified API works consistently across cloud providers
• Error handling and logging are standardized
• Provider factory enables runtime provider selection
• Documentation covers abstraction layer usage
• Unit tests validate all provider implementations
• Integration tests verify cross-cloud compatibility

Implementation Strategy

1. Interface Design: Define abstract base classes for cloud operations
2. GCP Migration: Refactor existing GCP code to use abstraction layer
3. AWS Implementation: Create AWS provider using the same interfaces
4. Configuration: Add provider-specific configuration management
5. Factory Pattern: Implement provider selection and initialization
6. Testing: Comprehensive testing across all providers

Key Benefits

• Maintainability: Single codebase supports multiple clouds
• Extensibility: Easy to add new cloud providers
• Consistency: Uniform behavior across different clouds
• Testing: Simplified mocking and testing strategies
• Migration: Smooth provider switching capabilities

Files to Create/Modify

• backend/cloud/ - New directory for cloud abstractions
• backend/cloud/providers/ - Provider-specific implementations
• backend/config.py - Add cloud provider configuration
• backend/storage_manager.py - Update to use abstraction layer
• Helm charts - Provider-agnostic configuration templates

Related to

Epic #22 - Multi-Cloud Support

Definition of Done

• Abstraction layer provides unified interface for cloud operations
• Both GCP and AWS providers are fully functional
• Existing functionality is preserved and enhanced
• Code is maintainable and well-documented
• Testing validates all provider implementations