API-DOCUMENTATION.md

📚 StepTwo Gallery Scraper - API Documentation

Overview

StepTwo Gallery Scraper provides a comprehensive API for programmatic gallery scraping, error handling, and performance monitoring. This documentation covers all public APIs and integration points.

Core APIs

🎯 Scraping API

RobustHelpers

Central utility class for robust web scraping operations.

// Image gathering with robust error handling
const images = await RobustHelpers.gatherImages({
  selectors: ['img', '.gallery-item img'],
  filters: {
    minWidth: 100,
    minHeight: 100,
    excludeDataURIs: true
  },
  retries: 3,
  timeout: 30000
});

// URL normalization
const normalizedUrl = RobustHelpers.normalizeURL(relativeUrl, baseUrl, {
  removeQueryParams: false,
  forceHTTPS: true
});

// Element utilities
const isVisible = RobustHelpers.isElementVisible(element);
const text = RobustHelpers.extractTextFromElement(element, { trim: true });

Adaptive Selector System

Advanced selector system with pattern recognition and fallback strategies.

// Generate adaptive selectors
const selectors = AdaptiveSelectorSystem.generateSelectors(element, {
  includeParent: true,
  includeAttributes: ['class', 'id', 'data-*'],
  maxDepth: 5
});

// Smart selector matching
const elements = AdaptiveSelectorSystem.smartQuery(document, 'img.gallery', {
  fallbackSelectors: ['img[src*="gallery"]', '.image-container img'],
  context: document.querySelector('.gallery-wrapper')
});

🚨 Error Handling API

ErrorHandlingSystem

Centralized error management with user-friendly reporting.

// Handle errors with context
StepTwoErrorHandler.handleError(
  new Error('Network timeout'),
  'Image Download',
  { url: 'https://example.com/image.jpg', attempt: 3 },
  'high' // severity
);

// Get error statistics
const stats = StepTwoErrorHandler.getErrorStats();
console.log('Total errors:', stats.totalErrors);
console.log('By severity:', stats.bySeverity);

// Clear error history
StepTwoErrorHandler.clearHistory();

⚡ Performance Monitoring API

PerformanceMonitoringSystem

Comprehensive performance tracking and optimization recommendations.

// Monitor operations
const operation = StepTwoPerformanceMonitor.startOperation(
  'gallery-scrape-001',
  'scraping',
  { galleryUrl: 'https://example.com', imageCount: 50 }
);

// Complete operation tracking
StepTwoPerformanceMonitor.endOperation('gallery-scrape-001', {
  success: true,
  imagesProcessed: 48,
  duration: 5430
});

// Get performance report
const report = StepTwoPerformanceMonitor.getPerformanceReport();
console.log('Operation stats:', report.operationStats);
console.log('Recommendations:', report.recommendations);

📦 Export System API

AdvancedExportSystem

Professional export capabilities with multiple formats.

// Export to Excel with advanced formatting
const excelData = await AdvancedExportSystem.exportToExcel(galleryData, {
  includeMetadata: true,
  includeThumbnails: true,
  includeStatistics: true,
  sheetName: 'Gallery Export'
});

// Export to comprehensive HTML report
const htmlReport = AdvancedExportSystem.generateHTMLReport(galleryData, {
  includePreview: true,
  optimizeForPrint: true,
  includeStatistics: true
});

// Create ZIP package with images and metadata
const zipPackage = await AdvancedExportSystem.createZIPPackage(galleryData, {
  includeImages: true,
  includeMetadata: true,
  compression: 'medium'
});

🔄 Background Processing API

BatchOperationsManager

Efficient batch processing with queue management.

// Add batch operation
BatchOperationsManager.addOperation({
  type: 'download',
  urls: imageUrls,
  options: {
    maxConcurrent: 3,
    retryOnFailure: true,
    timeout: 30000
  }
});

// Monitor batch progress
BatchOperationsManager.onProgress((progress) => {
  console.log(`Progress: ${progress.completed}/${progress.total}`);
});

// Handle batch completion
BatchOperationsManager.onComplete((results) => {
  console.log('Batch completed:', results.summary);
});

Chrome Extension APIs

📡 Message Passing

Background Script Communication

Communication between content scripts and background service worker.

// Send message from content script
chrome.runtime.sendMessage({
  action: 'startScraping',
  data: {
    selectors: ['img.gallery-image'],
    options: { includeMetadata: true }
  }
}, (response) => {
  console.log('Scraping started:', response.operationId);
});

// Handle messages in background script
chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
  switch (message.action) {
    case 'startScraping':
      const operationId = startScrapingOperation(message.data);
      sendResponse({ success: true, operationId });
      break;
    
    case 'getProgress':
      const progress = getOperationProgress(message.operationId);
      sendResponse({ progress });
      break;
  }
});

Storage API

Persistent storage for settings and cache.

// Save user preferences
await chrome.storage.sync.set({
  'userPreferences': {
    exportFormat: 'excel',
    includeMetadata: true,
    maxConcurrentDownloads: 3
  }
});

// Load user preferences
const { userPreferences } = await chrome.storage.sync.get('userPreferences');

// Local storage for cache
await chrome.storage.local.set({
  'galleryCache': {
    [galleryUrl]: {
      images: imageData,
      timestamp: Date.now(),
      metadata: galleryMetadata
    }
  }
});

🎨 UI Integration API

Popup Interface

Main extension popup interface.

// Update popup status
updatePopupStatus({
  status: 'active',
  message: 'Scraping in progress...',
  progress: 65,
  details: {
    found: 45,
    processed: 29,
    remaining: 16
  }
});

// Handle user actions
document.getElementById('start-scraping').addEventListener('click', async () => {
  const options = getScrapingOptions();
  const response = await chrome.runtime.sendMessage({
    action: 'startScraping',
    data: options
  });
  
  if (response.success) {
    showProgress(response.operationId);
  }
});

Dashboard Interface

Windowed dashboard for detailed operations.

// Open dashboard
openDashboard({
  mode: 'scraping',
  galleryUrl: currentUrl,
  initialData: detectedImages
});

// Update dashboard with real-time data
updateDashboard({
  operationId: 'gallery-scrape-001',
  progress: {
    completed: 25,
    total: 50,
    current: 'Processing image 26/50'
  },
  statistics: {
    successRate: 96,
    avgDownloadTime: 1200,
    totalSize: '15.2 MB'
  }
});

Integration Examples

🔧 Custom Site Integration

// Define custom site profile
const customSiteProfile = {
  name: 'Custom Gallery Site',
  domains: ['custom-gallery.com'],
  selectors: {
    images: 'img.custom-gallery-image',
    container: '.custom-gallery-container',
    pagination: '.custom-pagination a'
  },
  processing: {
    extractMetadata: (element) => ({
      title: element.getAttribute('data-title'),
      description: element.getAttribute('alt'),
      category: element.closest('[data-category]')?.dataset.category
    }),
    normalizeUrl: (url) => url.replace('/thumb/', '/full/')
  }
};

// Register custom profile
SiteProfileManager.registerProfile(customSiteProfile);

🧪 Testing Integration

// Mock extension APIs for testing
const mockChrome = {
  runtime: {
    sendMessage: jest.fn().mockResolvedValue({ success: true }),
    onMessage: {
      addListener: jest.fn()
    }
  },
  storage: {
    sync: {
      get: jest.fn().mockResolvedValue({}),
      set: jest.fn().mockResolvedValue()
    }
  }
};

global.chrome = mockChrome;

// Test scraping functionality
test('should scrape gallery images', async () => {
  const images = await RobustHelpers.gatherImages({
    selectors: ['img.test-image']
  });
  
  expect(images).toHaveLength(3);
  expect(images[0]).toHaveProperty('src');
  expect(images[0]).toHaveProperty('metadata');
});

Configuration

🛠️ Extension Configuration

{
  "scraping": {
    "maxConcurrentRequests": 3,
    "defaultTimeout": 30000,
    "retryAttempts": 3,
    "respectRobotsTxt": true
  },
  "export": {
    "defaultFormat": "excel",
    "includeMetadata": true,
    "includeStatistics": true,
    "compression": "medium"
  },
  "performance": {
    "enableMonitoring": true,
    "memoryTrackingInterval": 10000,
    "networkTrackingEnabled": true,
    "reportingInterval": 30000
  },
  "errorHandling": {
    "enableUserNotifications": true,
    "maxErrorHistory": 100,
    "enableErrorReporting": false
  }
}

🎯 Development Configuration

// Development mode settings
const developmentConfig = {
  debug: true,
  verbose: true,
  enableMockData: true,
  testGalleryUrl: 'http://localhost:3000/test-gallery.html',
  skipPermissionChecks: true
};

// Apply development configuration
if (process.env.NODE_ENV === 'development') {
  Object.assign(defaultConfig, developmentConfig);
}

Events & Hooks

📡 Event System

// Listen for scraping events
document.addEventListener('steptwo:scrapingStarted', (event) => {
  console.log('Scraping started:', event.detail);
});

document.addEventListener('steptwo:scrapingProgress', (event) => {
  updateProgressBar(event.detail.progress);
});

document.addEventListener('steptwo:scrapingCompleted', (event) => {
  handleScrapingResults(event.detail.results);
});

// Dispatch custom events
const customEvent = new CustomEvent('steptwo:customAction', {
  detail: { data: 'custom data' }
});
document.dispatchEvent(customEvent);

🔌 Lifecycle Hooks

// Hook into extension lifecycle
StepTwoExtension.onInit(() => {
  console.log('Extension initialized');
});

StepTwoExtension.onActivated((tab) => {
  console.log('Extension activated on tab:', tab.url);
});

StepTwoExtension.onDeactivated(() => {
  console.log('Extension deactivated');
  cleanupResources();
});

Error Codes & Status

🚨 Common Error Codes

Code	Type	Description	Resolution
ERR_NETWORK_TIMEOUT	Network	Request timeout	Increase timeout or check connection
ERR_INVALID_SELECTOR	Scraping	CSS selector invalid	Verify selector syntax
ERR_PERMISSION_DENIED	Security	Insufficient permissions	Check extension permissions
ERR_EXPORT_FAILED	Export	Export operation failed	Check export options and data
ERR_MEMORY_LIMIT	Performance	Memory usage too high	Reduce batch size or enable optimization

✅ Status Codes

Status	Description
IDLE	Extension ready for operation
SCANNING	Detecting gallery content
SCRAPING	Actively scraping images
PROCESSING	Processing scraped data
EXPORTING	Generating export files
COMPLETED	Operation completed successfully
ERROR	Operation failed with error

Best Practices

🎯 Performance Optimization

// Batch processing for better performance
const batchSize = 10;
const batches = chunkArray(imageUrls, batchSize);

for (const batch of batches) {
  await Promise.all(batch.map(url => processImage(url)));
  // Small delay between batches to prevent overwhelming
  await sleep(100);
}

// Memory management
const processLargeDataset = async (data) => {
  for (let i = 0; i < data.length; i += 100) {
    const chunk = data.slice(i, i + 100);
    await processChunk(chunk);
    
    // Force garbage collection hint
    if (global.gc) {
      global.gc();
    }
  }
};

🔒 Security Considerations

// Validate and sanitize URLs
const validateUrl = (url) => {
  try {
    const parsed = new URL(url);
    // Only allow http and https protocols
    if (!['http:', 'https:'].includes(parsed.protocol)) {
      throw new Error('Invalid protocol');
    }
    return parsed.href;
  } catch (error) {
    throw new Error('Invalid URL format');
  }
};

// Content Security Policy compliance
const createSecureElement = (tag, attributes = {}) => {
  const element = document.createElement(tag);
  
  // Sanitize attributes
  Object.entries(attributes).forEach(([key, value]) => {
    if (allowedAttributes.includes(key)) {
      element.setAttribute(key, sanitizeAttributeValue(value));
    }
  });
  
  return element;
};

📊 Monitoring & Analytics

// Performance monitoring
const monitorOperation = async (operationName, operation) => {
  const startTime = performance.now();
  const startMemory = performance.memory?.usedJSHeapSize || 0;
  
  try {
    const result = await operation();
    
    const endTime = performance.now();
    const endMemory = performance.memory?.usedJSHeapSize || 0;
    
    // Log performance metrics
    console.log(`Operation ${operationName}:`, {
      duration: `${(endTime - startTime).toFixed(2)}ms`,
      memoryDelta: `${((endMemory - startMemory) / 1024 / 1024).toFixed(2)}MB`
    });
    
    return result;
  } catch (error) {
    StepTwoErrorHandler.handleError(error, operationName, {}, 'high');
    throw error;
  }
};

Support & Troubleshooting

🐛 Common Issues

1. Images not loading: Check network connectivity and CORS policies
2. Slow performance: Reduce concurrent operations or increase timeout
3. Memory issues: Enable performance monitoring and optimize batch sizes
4. Export failures: Verify data format and available storage space

🔧 Debugging

// Enable debug mode
StepTwoExtension.setDebugMode(true);

// Access debug information
const debugInfo = StepTwoExtension.getDebugInfo();
console.log('Debug info:', debugInfo);

// Performance debugging
StepTwoPerformanceMonitor.captureMemorySnapshot('debug-checkpoint');
const report = StepTwoPerformanceMonitor.getPerformanceReport();
console.log('Performance report:', report);

For additional support, please refer to the GitHub Issues page.