Testing updates

tools/testing/README.md

@@ -0,0 +1,221 @@
+# Scopy Testing Tools
+
+A comprehensive set of Python tools for managing Scopy's manual testing workflow. These tools automate test environment setup, result parsing, and CSV template generation for release testing processes.
+
+## Quick Start
+
+### 1. **Generate Test Environment**
+```bash
+# Create test environment for v3.0.0 with all tests
+python3 setup_test_environment.py v3.0.0
+
+# Filter by component
+python3 setup_test_environment.py v3.0.0 --component adc dac
+
+# Filter by priority level
+python3 setup_test_environment.py v3.0.0 --rbp P0 P1
+
+# Release testing: P0 tests + new features since last version
+python3 setup_test_environment.py v3.0.0 --rbp P0 --new-since v2.5.0
+```
+
+### 2. **Execute Tests Manually**
+Edit the generated RST files in `testing_results/testing_results_v3.0.0/` and fill in:
+- **Tested OS:** Your operating system
+- **Comments:** Test execution notes
+- **Result:** PASS or FAIL
+
+### 3. **CSV Template Auto-Generated**
+The CSV template is automatically created during step 1. You can also generate it manually:
+```bash
+python3 parseTestResults.py v3.0.0
+```
+
+## Detailed Usage
+
+### `setup_test_environment.py` - Test Environment Setup
+
+**Purpose:** Creates filtered copies of test documentation for manual testing execution.
+
+**Basic Usage:**
+```bash
+python3 setup_test_environment.py <version> [OPTIONS]
+```
+
+**Parameters:**
+- `version` - Version name for the test directory (e.g., v3.0.0)
+- `--rbp` - Filter by RBP priority levels (P0, P1, P2, P3)
+- `--component` - Filter by component names (adc, dac, m2k, core, etc.)
+- `--new-since` - Include tests added since specified git version
+
+**Examples:**
+```bash
+# Copy all tests for v3.0.0
+python3 setup_test_environment.py v3.0.0
+
+# Only P0 and P1 priority tests
+python3 setup_test_environment.py v3.0.0 --rbp P0 P1
+
+# Only ADC and DAC component tests
+python3 setup_test_environment.py v3.0.0 --component adc dac
+
+# Combined filters: M2K component with P0 priority
+python3 setup_test_environment.py v3.0.0 --component m2k --rbp P0
+
+# Release testing: All P0 tests + new tests since v2.5.0
+python3 setup_test_environment.py v3.0.0 --rbp P0 --new-since v2.5.0
+
+# Only new features testing
+python3 setup_test_environment.py v3.0.0 --new-since v2.5.0
+```
+
+**Output:**
+- Creates `testing_results/testing_results_v3.0.0/` directory
+- Contains filtered RST files ready for manual testing
+- Preserves RST structure for documentation compilation
+
+### `parseTestResults.py` - CSV Template Generation
+
+**Purpose:** Generates CSV templates from testing directories for manual test execution tracking.
+
+**Basic Usage:**
+```bash
+python3 parseTestResults.py <version> [OPTIONS]
+```
+
+**Parameters:**
+- `version` - Version to parse (must match testing_results directory)
+- `--output` - Custom output Excel file path (optional)
+
+**Examples:**
+```bash
+# Parse v3.0.0 test results
+python3 parseTestResults.py v3.0.0
+
+# Custom output location
+python3 parseTestResults.py v3.0.0 --output /path/to/custom_results.xlsx
+```
+
+**Output:**
+- Creates `testing_results/testing_results_{version}.csv`
+- Template with empty result fields for manual completion
+- Columns: Test UID | Component | RBP | Result | Tested OS | Comments | Tester | File
+
+### `generateTestsTable.py` - Source Documentation Tracking
+
+**Purpose:** Generates Excel tracking tables directly from source documentation (before test execution).
+
+**Basic Usage:**
+```bash
+python3 generateTestsTable.py <version> [OPTIONS]
+```
+
+**Parameters:**
+- `version` - Version name for Excel worksheet
+- `--sort` - Sort tests by RBP priority instead of UID
+- `--filter` - Filter tests by RBP levels (e.g., P0,P1)
+
+**Examples:**
+```bash
+# Generate tracking table for v3.0.0
+python3 generateTestsTable.py v3.0.0
+
+# Only P0 and P1 tests, sorted by priority
+python3 generateTestsTable.py v3.0.0 --filter P0,P1 --sort
+```
+
+**Output:**
+- Creates/updates `scopy_tests_tracking.xlsx` in scopy root
+- Columns: Test UID | Test RBP | Test Pass/Fail (empty for manual entry)
+
+## Complete Testing Workflow
+
+### Release Testing Example (v3.0.0)
+
+**1. Prepare Test Environment**
+```bash
+# Create comprehensive release testing environment
+python3 setup_test_environment.py v3.0.0 --rbp P0 --new-since v2.5.0
+```
+
+**2. Execute Tests**
+```bash
+cd testing_results/testing_results_v3.0.0/
+# Manually edit RST files with test results:
+```
+
+**Example test result format:**
+```rst
+**Tested OS:** Windows 11
+
+**Comments:** Test passed successfully. Minor UI lag observed but within acceptable range.
+
+**Result:** PASS
+```
+
+**3. Track Test Execution**
+- Edit the auto-generated CSV template: `testing_results/testing_results_v3.0.0.csv`
+- Fill in Result, Tested OS, Comments, and Tester columns for each test
+
+**4. Review Results**
+- Open `testing_results/testing_results_v3.0.0.csv`
+- Use spreadsheet software for analysis and reporting
+
+
+##  Requirements
+
+### Git Requirements
+- Git repository (required for `--new-since` functionality)
+- Proper git tags for version references
+
+### File Structure Requirements
+```
+scopy/
+├── docs/tests/           # Source test documentation
+├── testing_results/      # Generated test environments
+└── tools/testing/        # This toolset
+```
+
+## CSV Template Format
+
+### Columns
+- **Test UID** - Unique test identifier
+- **Component** - Test component (adc, dac, m2k, core, etc.)
+- **RBP** - Risk-based priority (P0, P1, P2, P3)
+- **Result** - Test outcome (empty template field for PASSED/FAILED/SKIPPED)
+- **Tested OS** - Operating system used for testing (empty template field)
+- **Comments** - Test execution notes (empty template field)
+- **Tester** - Person executing the tests (empty template field)
+- **File** - Source RST filename
+
+### Multiple Versions
+- Each version gets its own CSV file: `testing_results_{version}.csv`
+- Historical test data preserved across versions
+- Easy import into spreadsheet tools for analysis
+
+## Troubleshooting
+
+### Common Issues
+
+**"Git tag not found" error:**
+```bash
+# Check available git tags
+git tag -l
+
+# Use exact tag name
+python3 setup_test_environment.py v3.0.0 --new-since v2.5.0
+```
+
+**"Not in a git repository" error:**
+```bash
+# Ensure you're in the scopy git repository root
+cd /path/to/scopy
+python3 tools/testing/setup_test_environment.py v3.0.0 --new-since v2.5.0
+```
+
+**No tests found:**
+- Verify `docs/tests/` directory exists
+- Check test files contain proper `**UID:**` and `**RBP:**` metadata
+
+
+

tools/testing/file_filters.py

@@ -0,0 +1,80 @@
+#!/usr/bin/env python3
+"""
+File Filters Module
+Handles file and content filtering logic for test environment setup.
+
+This module provides functionality for:
+- Component-based file filtering
+- RBP-based content filtering
+- New test detection and filtering
+- Helper functions for filtering decisions
+"""
+
+from rst_parser import parse_rst_structure, rebuild_rst_file
+
+
+def _should_include_file(test_file, component_filter, new_uids):
+    """Determine if file should be included based on filters."""
+    # Skip all index files - they're not needed for CSV workflow
+    if 'index.rst' in test_file['relative_path']:
+        return False
+
+    # Include files from specified components
+    if test_file['component'] in component_filter:
+        return True
+
+    # Include files containing new tests (bypass component filter)
+    if new_uids and any(test['uid'] in new_uids for test in test_file['tests']):
+        return True
+
+    return False
+
+
+def filter_by_component(test_files, component_filter, new_uids=None):
+    """Simple file-level filtering by component, with new-since support."""
+    if not component_filter:
+        return test_files
+
+    return [tf for tf in test_files
+            if _should_include_file(tf, component_filter, new_uids)]
+
+
+def _should_include_test(test, rbp_filter, new_uids):
+    """Determine if test should be included based on filters."""
+    # Include if it's a new test (always)
+    if new_uids and test['uid'] in new_uids:
+        return True
+
+    # Include if matches RBP filter
+    if rbp_filter and test['rbp'] in rbp_filter:
+        return True
+
+    # Include if no filters specified
+    if not rbp_filter and not new_uids:
+        return True
+
+    return False
+
+
+def _extract_all_uids(test_files):
+    """Extract all UIDs from test files efficiently."""
+    return {test['uid'] for file_info in test_files
+            for test in file_info['tests']
+            if test['uid'] != 'MISSING'}
+
+
+def filter_rst_content(file_path, rbp_filter, new_uids=None):
+    """Parse RST file and keep only tests matching RBP filter or new tests."""
+
+    with open(file_path, 'r', encoding='utf-8') as f:
+        content = f.read()
+
+    # Split into header + test sections
+    structure = parse_rst_structure(content)
+
+    # Filter tests by RBP or new-since
+    filtered_tests = [test for test in structure['tests']
+                     if _should_include_test(test, rbp_filter, new_uids)]
+
+    # Reconstruct file
+    return rebuild_rst_file(structure['header'], filtered_tests)

tools/testing/file_operations.py

@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+"""
+File Operations Module
+Handles file copying operations and index file management.
+
+This module provides functionality for:
+- Copying test files with or without modification
+- Managing index file toctree entries
+- Component tracking for copied files
+- RST content-level filtering and file operations
+"""
+
+import os
+import shutil
+from file_filters import filter_rst_content
+
+
+def copy_files_unchanged(test_files, dest_dir, component_filter=None):
+    """Copy files without content modification."""
+    copied_count = 0
+
+    for test_file in test_files:
+        src_path = test_file['file_path']
+        dest_path = os.path.join(dest_dir, test_file['relative_path'])
+
+        # Create destination directory
+        os.makedirs(os.path.dirname(dest_path), exist_ok=True)
+
+        # Copy file unchanged
+        shutil.copy2(src_path, dest_path)
+        copied_count += 1
+
+    return copied_count
+
+
+def filter_by_rbp_content(test_files, rbp_filter, dest_dir, component_filter=None, new_uids=None):
+    """Content-level filtering: modify files to keep only specified RBP tests or new tests."""
+    if not rbp_filter and not new_uids:
+        # No filters - copy files as-is
+        return copy_files_unchanged(test_files, dest_dir, component_filter)
+
+    copied_count = 0
+
+    # Filter active - parse and modify content
+    for test_file in test_files:
+        dest_path = os.path.join(dest_dir, test_file['relative_path'])
+
+        # Create destination directory
+        os.makedirs(os.path.dirname(dest_path), exist_ok=True)
+
+        if test_file['tests']:  # File contains tests
+            filtered_content = filter_rst_content(test_file['file_path'], rbp_filter, new_uids)
+            with open(dest_path, 'w', encoding='utf-8') as f:
+                f.write(filtered_content)
+        else:  # File has no tests (docs, etc.)
+            shutil.copy2(test_file['file_path'], dest_path)
+
+        copied_count += 1
+
+    return copied_count