Full Localization Pipeline: Capture → Persist → Generate Previews → Submit to App Store

[ebowwa]

Overview

Complete localization pipeline using app-store-connect-wrapper to discover supported languages, capture localized screenshots, persist them for reuse, generate App Store previews, and ultimately submit to App Store Connect.

🔴 CRITICAL BUGS TO FIX FIRST (from #11)

Data Loss Risk - Screenshots NOT Auto-Saved

• CRITICAL: Screenshots only stored in memory, lost when app closes
• IMPACT: Hours of work can be lost with no recovery
• LOCATION: ScreenshotManager.swift:276-350 - export functions exist but NOT auto-called
• FIX REQUIRED: Implement auto-save immediately after capture

Recording-Screenshot Synchronization Missing

• PROBLEM: No linkage between recordings and screenshots
• CANNOT: Guarantee same screenshots captured at same points across locales
• MISSING: Screenshot markers in recordings
• MISSING: Locale information stored with recordings

Current Persistence State

Component	Auto-Save	Risk Level	Location
Screenshots	❌ NO	HIGH - Data loss on app close	Memory only
Recordings	✅ YES	LOW	~/Documents/recordings.json
Scenarios	✅ YES	LOW	UserDefaults
Locales	✅ YES	LOW	UserDefaults

The Full Pipeline Vision (Enhanced with #7, #9, #11)

1. DISCOVER (via App Store Connect API)
   ↓
2. CAPTURE with AI Vision (#7)
   - AI analyzes screens in real-time
   - Identifies optimal screenshot points
   - Records actions with screenshot markers
   ↓
3. PERSIST with Training Pipeline (#9)
   - Auto-save screenshots immediately
   - Build training dataset from interactions
   - Store recording-screenshot correlations
   ↓
4. GENERATE (create App Store previews from persisted screenshots)
   ↓
5. SUBMIT (upload previews to App Store Connect)

Purpose

Build a complete automated pipeline that:

• Discovers what localizations an app supports
• Captures screenshots across all languages automatically with AI assistance
• Persists screenshots in organized storage for future use AND training data
• Generates App Store preview videos/images from persisted screenshots
• Submits final previews to App Store Connect

1. Discovery & Planning Phase

# discover_and_plan.py
from app_store_connect_wrapper import AppStoreConnect

class LocalizationPipeline:
    def __init__(self, bundle_id):
        self.asc = AppStoreConnect()
        self.app = self.asc.get_app_info(bundle_id)
        self.localizations = self.app.localizations
        
    def create_capture_plan(self):
        """Generate screenshot capture plan for all locales"""
        return {
            "app": self.app.bundle_id,
            "version": self.app.version,
            "localizations": self.localizations,
            "required_screenshots": {
                "iPhone_6_7": ["screen1", "screen2", "screen3", "screen4", "screen5"],
                "iPhone_6_5": ["screen1", "screen2", "screen3", "screen4", "screen5"],
                "iPad_12_9": ["screen1", "screen2", "screen3", "screen4", "screen5"]
            },
            "preview_requirements": {
                "format": "video",  # or "static_images"
                "duration": 30,     # seconds for video
                "highlight_features": ["feature1", "feature2", "feature3"]
            }
        }

2. AI-Powered Capture & Persistence Layer (Incorporating #7 & #9)

AI Vision Integration (from #7)

// AI-powered screen analysis and action execution
class AIVisionController {
    let visionModel: VisionLanguageModel // Gemini, Claude, or GPT-4V
    
    func analyzeScreen(_ screenshot: NSImage) -> ScreenAnalysis {
        // Send screenshot to VLM for analysis
        let analysis = visionModel.analyze(screenshot)
        return ScreenAnalysis(
            uiElements: analysis.detectedElements,
            optimalScreenshotPoint: analysis.isKeyScreen,
            suggestedActions: analysis.nextActions,
            layoutIssues: analysis.potentialProblems
        )
    }
    
    func executeIntelligentAction(intent: String, context: ScreenContext) {
        // AI determines best action based on intent and current screen
        let action = visionModel.determineAction(intent, context)
        actionRecorder.execute(action)
    }
}

Critical Fix: Auto-Save Implementation (from #11)

// In ScreenshotManager.swift - MUST IMPLEMENT IMMEDIATELY
class ScreenshotPersistenceManager {
    let basePath = "~/AppStoreAssets/{BundleID}/{Version}/"
    
    struct PersistedScreenshot {
        let locale: String
        let deviceType: String
        let screenName: String
        let filePath: String
        let capturedAt: Date
        let metadata: [String: Any]  // OCR text, UI elements detected, etc.
        let linkedActionIndex: Int?   // Links to recording action
    }
    
    func captureScreenshot(...) {
        // ... existing capture code ...
        
        // CRITICAL FIX: Auto-save immediately
        self.autoSaveScreenshot(screenshot)
        
        // Link to current recording if active
        if actionRecorder.isRecording {
            screenshot.linkedActionIndex = actionRecorder.currentActions.count
            actionRecorder.addScreenshotMarker(screenshot.id)
        }
    }
    
    private func autoSaveScreenshot(_ screenshot: Screenshot) {
        let sessionPath = getCurrentSessionPath()
        let screenshotPath = sessionPath
            .appendingPathComponent("screenshots")
            .appendingPathComponent(screenshot.locale.code)
            .appendingPathComponent("\(screenshot.timestamp).png")
        
        // Save to disk IMMEDIATELY - don't wait for export
        saveImage(screenshot.image, to: screenshotPath)
        
        // Save metadata for AI training (from #9)
        let metadata = [
            "locale": screenshot.locale,
            "device": screenshot.deviceType,
            "screen_name": screenshot.scenario.name,
            "ui_elements": detectUIElements(screenshot),
            "timestamp": screenshot.timestamp,
            "linked_action": screenshot.linkedActionIndex,
            "ai_analysis": AIVisionController.analyzeScreen(screenshot.image)
        ]
        
        saveMetadata(metadata, path: screenshotPath + ".json")
        
        // Add to training dataset (from #9)
        TrainingDataCollector.shared.addScreenshot(screenshot, metadata)
    }
}

Recording-Screenshot Integration (Critical Missing Feature from #11)

// In ActionRecorder.swift
struct Recording: Codable {
    let id: UUID
    var name: String
    let windowBounds: CGRect
    var actions: [RecordedAction]
    let recordedAt: Date
    var annotations: [Int: String]
    var screenshotMarkers: [ScreenshotMarker] // NEW: Links to screenshots
    let locale: LocaleInfo // NEW: Store locale with recording
}

struct ScreenshotMarker: Codable {
    let actionIndex: Int
    let screenshotId: UUID
    let timestamp: TimeInterval
    let description: String
    let isRequired: Bool // For validation during replay
}

// Enhanced replay with auto-screenshot
func replay(_ recording: Recording, in windowBounds: CGRect, style: ReplayStyle) async {
    for (index, action) in recording.actions.enumerated() {
        // Check if this action has a screenshot marker
        if let marker = recording.screenshotMarkers.first(where: { $0.actionIndex == index }) {
            // Auto-capture screenshot at marked point
            screenshotManager.captureAtMarker(marker)
        }
        
        await executeAction(action, in: currentWindow.bounds)
    }
    
    // Validate all required screenshots were captured
    validateScreenshotCompleteness(recording.screenshotMarkers)
}

3. Training Data Pipeline (from #9)

// Recording Analytics and Model Training Data Collection
class TrainingDataCollector {
    static let shared = TrainingDataCollector()
    
    struct InteractionPattern {
        let sequence: [RecordedAction]
        let outcome: InteractionOutcome
        let context: AppContext
        let performance: PerformanceMetrics
    }
    
    func collectTrainingData(from recording: Recording, screenshots: [Screenshot]) {
        // Extract patterns for AI model training
        let patterns = extractPatterns(recording.actions)
        
        // Analyze user behavior
        let behavior = UserBehaviorAnalyzer.analyze(patterns)
        
        // Store for model training
        let trainingData = TrainingDataset(
            recordings: [recording],
            screenshots: screenshots,
            patterns: patterns,
            behavior: behavior,
            locale: recording.locale
        )
        
        // Persist to training pipeline
        saveToTrainingPipeline(trainingData)
    }
    
    func exportForModelTraining() -> URL {
        // Export collected data in format suitable for VLM training
        let dataset = prepareDataset()
        return dataset.export(format: .coreML) // or .tensorflow, .pytorch
    }
}

4. Preview Generation from Persisted Screenshots

# generate_previews.py - Enhanced with AI insights
import cv2
from pathlib import Path

class PreviewGenerator:
    def __init__(self, app_id, version):
        self.screenshot_base = f"~/AppStoreAssets/{app_id}/{version}/"
        self.ai_analyzer = AIScreenAnalyzer()
        
    def generate_app_preview(self, locale, device_type):
        """Generate App Store preview video from persisted screenshots"""
        
        # Load persisted screenshots with their AI analysis
        screenshots = self.load_screenshots_with_metadata(locale, device_type)
        
        # AI selects best screenshots for preview
        selected = self.ai_analyzer.select_optimal_screenshots(screenshots, 
            criteria=["visual_appeal", "feature_showcase", "user_flow"])
        
        # Create preview video with AI-suggested transitions
        preview = self.create_video_preview(selected, 
            self.ai_analyzer.suggest_transitions(selected))
        
        # Add localized captions from AI analysis
        preview_with_captions = self.add_ai_generated_captions(preview, locale)
        
        # Export in App Store format
        output_path = f"{self.screenshot_base}/{locale}/{device_type}/preview.mp4"
        self.export_for_app_store(preview_with_captions, output_path)
        
        return output_path

5. Complete Workflow Example (Updated)

# 1. Discover localizations
python3 discover_localizations.py --app com.myapp
# Output: Found 30 localizations

# 2. Record base flow with AI assistance and screenshot markers
# In iOSTestingGym:
# - Start recording
# - AI suggests optimal screenshot points
# - Screenshots auto-saved immediately (no data loss\!)
# - Recording saves with screenshot markers and locale

# 3. Automated replay across all locales
for locale in locales:
    # Switch iPhone to locale
    # Load recording WITH screenshot markers
    # Replay automatically captures at marked points
    # All screenshots auto-persisted

# 4. Generate previews from persisted screenshots
python3 generate_previews.py --app com.myapp --version 1.2.0
# AI selects best screenshots
# Creates 90 preview videos (30 locales × 3 device types)

# 5. Submit to App Store
python3 submit_to_app_store.py --app com.myapp
# Uploads all previews to App Store Connect

Data Persistence Structure (Enhanced)

~/AppStoreAssets/
├── com.myapp/
│   ├── 1.2.0/
│   │   ├── sessions/
│   │   │   ├── 2024-01-15_14-30-00/
│   │   │   │   ├── recording.json (with screenshot markers & locale)
│   │   │   │   ├── screenshot_manifest.json
│   │   │   │   └── training_data.json
│   │   ├── en-US/
│   │   │   ├── iPhone_6_7/
│   │   │   │   ├── screenshots/
│   │   │   │   │   ├── home_2024-01-15.png
│   │   │   │   │   ├── home_2024-01-15.json (metadata + AI analysis)
│   │   │   │   │   └── ...
│   │   │   │   ├── recordings/
│   │   │   │   │   └── base_flow.json (with markers)
│   │   │   │   └── preview.mp4 (AI-generated)
│   │   │   ├── iPhone_6_5/
│   │   │   └── iPad_12_9/
│   │   ├── es-ES/
│   │   ├── fr-FR/
│   │   ├── training_data/
│   │   │   ├── interaction_patterns.json
│   │   │   ├── user_behaviors.json
│   │   │   └── model_training_export.coreml
│   │   └── ... (28 more locales)
│   └── 1.1.0/ (previous version kept for rollback)

Implementation Priority (CRITICAL PATH)

Phase 0: EMERGENCY FIXES (TODAY!)

• FIX CRITICAL BUG: Implement screenshot auto-save
• Add warning dialog: Alert before closing app if unsaved data
• Emergency backup: Auto-export to temp folder as safety net
• Fix visual feedback crashes: Comment out or fix EXC_BAD_ACCESS

Phase 1: Core Integration (This Week)

• Add screenshot markers to recordings
• Store locale with recordings
• Link screenshots to recording actions
• Implement replay with auto-capture at markers

Phase 2: AI Integration (from #7)

• Integrate Vision Language Model (Gemini/Claude/GPT-4V)
• AI-powered screenshot point detection
• Intelligent action execution
• Layout issue detection across locales

Phase 3: Training Pipeline (from #9)

• Collect interaction patterns
• Build training dataset structure
• Export for model training
• Performance analytics dashboard

Phase 4: Full Automation

• Automated locale switching
• Batch replay across all locales
• AI-optimized preview generation
• Direct App Store Connect submission

Success Criteria

• ZERO data loss - all screenshots auto-saved
• 100% correlation - every screenshot linked to recording action
• Complete validation - verify all screenshots captured across locales
• AI optimization - intelligent screenshot selection
• Training data - continuous model improvement
• < 1 hour - complete 30-language submission

Key Benefits

1. No Data Loss: Auto-save prevents losing hours of work
2. Perfect Sync: Screenshots and recordings perfectly correlated
3. AI Intelligence: Optimal screenshot selection and action execution
4. Continuous Learning: Training data improves system over time
5. Complete Automation: From capture to App Store submission

Immediate Workaround (Until Fixed)

CRITICAL: After EVERY screenshot session:

1. Export immediately - Don't do anything else first
2. Verify export - Check files exist on disk
3. Document session - Note which screenshots match which recording points
4. Backup recording - Copy recordings.json to safe location

Recording Workflow:

1. Start recording in base locale
2. Note timestamp at each screenshot point
3. Stop recording (auto-saves)
4. Take screenshots at noted points
5. EXPORT IMMEDIATELY before anything else
6. Document correlation manually

Replay Workflow:

1. Change iPhone locale
2. Load recording
3. Manually pause at documented points
4. Take screenshot
5. EXPORT AFTER EACH LOCALE
6. Repeat for all locales

Related Issues (Incorporated)

• Implement AI-Powered Control Mode with Vision Language Model Integration #7: AI-Powered Control Mode ✓ (integrated as AI Vision layer)
• Implement Data Persistence and Model Training Pipeline for Recording Analytics #9: Data Persistence and Model Training ✓ (integrated as training pipeline)
• Critical: Screenshot persistence and action-to-screenshot synchronization issues for localization testing #11: Screenshot persistence issues ✓ (critical fixes in Phase 0)

This represents the complete vision for a professional-grade iOS testing and localization system with AI intelligence and zero data loss.

[ebowwa]

This issue now consolidates and incorporates all critical details from:

• Implement AI-Powered Control Mode with Vision Language Model Integration #7: AI-Powered Control Mode with Vision Language Model Integration
• Implement Data Persistence and Model Training Pipeline for Recording Analytics #9: Data Persistence and Model Training Pipeline for Recording Analytics
• Critical: Screenshot persistence and action-to-screenshot synchronization issues for localization testing #11: Critical Screenshot Persistence and Action-to-Screenshot Synchronization Issues

All features and fixes from these related issues have been integrated into this comprehensive pipeline plan.