Testing: Enhance thread safety test assertions and diagnostics

[m-marinucci]

Context

As identified in PR #89 review, the thread safety tests need improvements:

1. Add explicit assertions for expected state after each cycle
2. Make the 95% threshold configurable
3. Report detailed diagnostics on failures

Current Issues

1. Missing State Assertions

Current code:

void run_repeated_initialization_test(int cycles) {
    total_threads_started++;
    // Missing assertions for system state
}

2. Hardcoded Failure Tolerance

bool test_passed = (failed_inits.load() == 0) && 
                  (operator_test_failures.load() == 0) &&
                  (successful_inits.load() >= config.num_threads * 0.95);

3. Limited Failure Diagnostics

No detailed reporting of which threads failed or why.

Proposed Improvements

1. Add State Validation Functions

class SystemStateValidator {
    static bool assert_system_clean() {
        // Check no dangling pointers
        // Verify all resources released
        // Confirm no leaked handles
        return true;
    }
    
    static bool assert_no_resource_leaks() {
        // Check memory usage
        // Verify file handles closed
        // Confirm thread count normal
        return true;
    }
    
    static bool validate_operator_registry() {
        // Verify operators properly registered
        // Check for duplicates
        // Validate consistency
        return true;
    }
};

void run_repeated_initialization_test(int cycles) {
    total_threads_started++;
    
    for (int i = 0; i < cycles; ++i) {
        // Initialize system
        initialize_system();
        
        // Validate state after initialization
        ASSERT_TRUE(SystemStateValidator::assert_system_clean());
        ASSERT_TRUE(SystemStateValidator::validate_operator_registry());
        
        // Run tests
        run_tests();
        
        // Cleanup
        cleanup_system();
        
        // Validate clean shutdown
        ASSERT_TRUE(SystemStateValidator::assert_no_resource_leaks());
    }
}

2. Configurable Thresholds

struct TestConfig {
    int num_threads = 16;
    int test_duration_ms = 5000;
    double success_threshold = 0.95;  // Configurable
    bool strict_mode = false;         // 100% success required
};

bool evaluate_test_results(const TestConfig& config) {
    if (config.strict_mode) {
        return failed_inits.load() == 0 && 
               operator_test_failures.load() == 0;
    }
    
    double success_rate = (double)successful_inits.load() / 
                         config.num_threads;
    return success_rate >= config.success_threshold;
}

3. Detailed Failure Reporting

struct ThreadResult {
    std::thread::id thread_id;
    bool success;
    std::string error_message;
    std::chrono::milliseconds duration;
    std::vector<std::string> stack_trace;
};

class TestReporter {
    std::vector<ThreadResult> results_;
    
public:
    void record_result(ThreadResult result) {
        std::lock_guard<std::mutex> lock(mutex_);
        results_.push_back(std::move(result));
    }
    
    void generate_report() {
        std::cout << "\n=== Detailed Test Report ===\n";
        
        // Summary statistics
        int passed = 0, failed = 0;
        for (const auto& r : results_) {
            if (r.success) passed++; else failed++;
        }
        
        std::cout << "Total: " << results_.size() 
                  << " | Passed: " << passed 
                  << " | Failed: " << failed << "\n\n";
        
        // Detailed failures
        if (failed > 0) {
            std::cout << "Failed Thread Details:\n";
            for (const auto& r : results_) {
                if (!r.success) {
                    std::cout << "Thread " << r.thread_id << ":\n"
                              << "  Error: " << r.error_message << "\n"
                              << "  Duration: " << r.duration.count() << "ms\n";
                    
                    if (!r.stack_trace.empty()) {
                        std::cout << "  Stack trace:\n";
                        for (const auto& frame : r.stack_trace) {
                            std::cout << "    " << frame << "\n";
                        }
                    }
                    std::cout << "\n";
                }
            }
        }
    }
};

Command Line Interface

# Configure success threshold
./thread_safety_test --threads 16 --threshold 0.95

# Strict mode (100% success required)
./thread_safety_test --threads 16 --strict

# Verbose diagnostics
./thread_safety_test --threads 16 --verbose --report-failures

Benefits

• Better debugging of intermittent failures
• Configurable for different environments
• Clear indication of system state issues
• Detailed failure analysis
• CI/CD friendly reporting

Related: PR #89