Skip to content

README.md

Latest commit

 

History

History
1424 lines (1076 loc) · 45.3 KB

README.md

File metadata and controls

1424 lines (1076 loc) · 45.3 KB
Raw

AFDD - Automated Fundamental Data Download

Python License Status Framework Adapters

Multi-website financial data download framework with intelligent browser automation, plugin architecture, and configurable adapters for Tijori Finance, Screener.in, and more.

Features

🌐 Multi-Website Support

  • Plugin Architecture: Extensible adapter system for multiple financial websites
  • Dynamic Configuration: Website-specific profiles with YAML configuration
  • Template Engine: Flexible URL construction with transformation pipelines
  • Fallback Strategies: Try multiple websites when one fails

🚀 Core Capabilities

  • Bulk Downloads: Process hundreds of stocks automatically
  • Smart Detection: Intelligent element detection with fallback selectors
  • Fast Processing: Optimized file detection (no 60-second timeouts)
  • Smart Naming: Automatic file renaming to standardized format
  • Resume Support: Continue interrupted downloads seamlessly
  • Multiple Formats: Support for Excel, CSV, and other file formats

🎯 Intelligent Discovery

  • URL Discovery: Template-based company page discovery with caching
  • Interactive Mode: User prompts for failed URL discovery attempts
  • Report-Based Retry: Process failed stocks from previous reports
  • CSV Sanitization: Convert URLs to proper company names
  • Rate Limiting: Respectful request timing per website requirements

📊 Reporting & Analysis

  • Comprehensive Reports: Detailed success/failure reporting
  • Cross-Website Comparison: Compare data from multiple sources
  • Error Handling: Robust error handling with retry mechanisms
  • Performance Metrics: Detailed timing and success rate statistics

🔧 Developer Features

  • Configurable Selectors: Dynamic element location strategies
  • Custom Adapters: Easy creation of new website adapters
  • Profile System: Website-specific configuration management
  • Backward Compatible: Existing workflows continue to work

Quick Start

# 1. Install AFDD
conda env create -f environment.yml
conda activate afdd
pip install -e .

# 2. See available websites
afdd --list-websites

# 3. Prepare your stock list (CSV or Excel with 'symbol' and 'name' columns)
# symbol,name
# RELIANCE,Reliance Industries Limited
# TCS,Tata Consultancy Services Limited

# 4. Choose your website and run AFDD
# Tijori Finance (default, premium data)
afdd --input my_stocks.csv --output ./downloads/ --resume --verbose

# Screener.in (free platform)
afdd --input my_stocks.csv --website screener --output ./downloads/ --verbose

# Interactive mode for handling failures
afdd --input my_stocks.csv --website tijori_finance --interactive --verbose

🌐 Multi-Website Examples

# Compare data from multiple sources
afdd --input stocks.csv --website tijori_finance --output ./tijori_data/
afdd --input stocks.csv --website screener --output ./screener_data/

# Use fallback strategy (try Tijori, then Screener for failures)
afdd --input stocks.csv --website tijori_finance --interactive --output ./downloads/
afdd --retry-from-report ./downloads/afdd_report_*.txt --website screener

# List all available websites
afdd --list-websites

Installation

Method 1: Automated Installation (Recommended)

AFDD includes automated installation scripts for both Windows and Mac/Linux that handle environment setup, dependencies, and testing.

🪟 Windows Installation

# Clone repository
git clone https://github.com/username/afdd.git
cd afdd

# Run automated Windows installer
install_conda.bat

🍎 Mac/Linux Installation

# Clone repository
git clone https://github.com/username/afdd.git
cd afdd

# Run automated installer
chmod +x install_conda.sh
./install_conda.sh

Both scripts will:

  • ✅ Check conda installation
  • 📦 Create conda environment from environment.yml
  • 🔧 Install AFDD package in development mode
  • 🌐 Setup ChromeDriver automatically
  • 🧪 Test all dependencies and imports
  • 📁 Create necessary directory structure
  • 🎯 Provide usage examples for multi-website framework

Method 2: Manual Installation

Conda Environment (Recommended)

# Clone repository
git clone https://github.com/username/afdd.git
cd afdd

# Create conda environment with all dependencies
conda env create -f environment.yml
conda activate afdd

# Install AFDD in development mode
pip install -e .

Pip + Virtual Environment

# Clone repository
git clone https://github.com/username/afdd.git
cd afdd

# Create virtual environment
python -m venv afdd_env
source afdd_env/bin/activate  # On Windows: afdd_env\Scripts\activate

# Install dependencies
pip install -r requirements.txt
pip install -e .

🔧 Post-Installation Verification

After installation, verify everything works:

# Activate environment
conda activate afdd  # or source afdd_env/bin/activate

# Test AFDD installation
afdd --version
afdd --list-websites

# Verify core functionality
python -c "from afdd.utils import system_health_check; system_health_check()"

📋 Prerequisites

  • Python 3.8+
  • Google Chrome browser
  • ChromeDriver (automatically managed with webdriver-manager)
  • Website Account (depends on chosen website):
    • Tijori Finance: Premium account required
    • Screener.in: Free access available (rate limited)

🚨 Troubleshooting Installation

Common Issues:

Conda not found:

# Install Miniconda first
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

ChromeDriver issues:

# Manual ChromeDriver installation
pip install webdriver-manager
python -c "from webdriver_manager.chrome import ChromeDriverManager; ChromeDriverManager().install()"

Permission errors on Mac/Linux:

chmod +x install_conda.sh
sudo ./install_conda.sh  # If needed

AFDD command not found:

# Use module syntax instead
python -m afdd.main --help

Website Support

AFDD now supports multiple financial data websites through a dynamic adapter system. Each website has its own configuration profile and specialized adapter.

📊 Available Websites

Website Key Authentication Data Format Best For
Tijori Finance tijori_finance Premium account required Excel (5 sheets) Comprehensive fundamental data
Screener.in screener Optional (free with limits) CSV Quick screening & ratios

🚀 Quick Website Selection

# List all available websites with descriptions
afdd --list-websites

# Use specific website
afdd --input stocks.csv --website tijori_finance    # Premium data
afdd --input stocks.csv --website screener          # Free screening data

🔄 Multi-Website Workflows

# Strategy 1: Primary + Fallback
afdd --input stocks.csv --website tijori_finance --interactive --output ./primary/
afdd --retry-from-report ./primary/afdd_report_*.txt --website screener --output ./fallback/

# Strategy 2: Comparative Analysis  
afdd --input stocks.csv --website tijori_finance --output ./tijori_data/
afdd --input stocks.csv --website screener --output ./screener_data/
# Compare results from both sources

# Strategy 3: Cost Optimization
afdd --input stocks.csv --website screener --output ./free_data/
# Use free data first, then premium for specific stocks

⚙️ Website-Specific Features

Tijori Finance

  • Authentication: Interactive login with browser
  • Data Quality: Premium fundamental data with 5 Excel sheets
  • Features: Full balance sheet, P&L, cash flow, ratios, quarterly results
  • Rate Limiting: 2-second delay (respectful)
  • Success Rate: ~95% with interactive mode

Screener.in

  • Authentication: Optional (works without login)
  • Data Quality: Free screening data in CSV format
  • Features: Key financial metrics and ratios
  • Rate Limiting: 3-second delay (free service)
  • Success Rate: ~85% (simpler pages)

🎯 Choosing the Right Website

Use Tijori Finance when:

  • ✅ You need comprehensive fundamental data
  • ✅ You have a premium account
  • ✅ Data quality is critical
  • ✅ You need historical quarterly data

Use Screener.in when:

  • ✅ You want free access to basic data
  • ✅ You're doing initial stock screening
  • ✅ You need quick ratio analysis
  • ✅ Budget is a constraint

Usage

🌐 Multi-Website Usage

Website Selection & Discovery

# Discover available websites
afdd --list-websites

# Basic usage with default website (tijori_finance)
afdd --input stocks.csv --output ./downloads/ --resume --verbose

# Explicit website selection
afdd --input stocks.csv --website tijori_finance --output ./downloads/ --verbose
afdd --input stocks.csv --website screener --output ./downloads/ --verbose

# Interactive mode for handling difficult stocks
afdd --input stocks.csv --website tijori_finance --interactive --verbose

Cross-Website Workflows

# Multi-source data collection
afdd --input stocks.csv --website tijori_finance --output ./tijori_data/ --verbose
afdd --input stocks.csv --website screener --output ./screener_data/ --verbose

# Fallback strategy: try premium first, then free
afdd --input stocks.csv --website tijori_finance --interactive --output ./downloads/
afdd --retry-from-report ./downloads/afdd_report_*.txt --website screener --output ./downloads/

# Cost optimization: start with free data
afdd --input stocks.csv --website screener --output ./free_data/ --verbose
# Then get premium data for specific stocks only

📊 Basic Usage Examples (Backward Compatible)

# Standard download (defaults to tijori_finance)
afdd --input stocks.csv --output ./downloads/

# Recommended for production use
afdd --input stocks.csv --output ./downloads/ --resume --verbose

# Custom settings with interactive mode
afdd --input fo_stocks.xlsx --output ./fo_data/ --delay 3 --batch-size 25 --resume --verbose --interactive

Complete CLI Reference & Use Cases

1. Standard Download Operations

# Basic download (defaults to tijori_finance)
afdd --input stocks.csv

# Production download (recommended) 
afdd --input stocks.csv --output ./downloads/ --resume --verbose

# Website-specific downloads
afdd --input stocks.csv --website tijori_finance --output ./downloads/ --resume --verbose
afdd --input stocks.csv --website screener --output ./downloads/ --resume --verbose

# Large dataset processing with website selection
afdd --input large_list.xlsx --website tijori_finance --output ./bulk_data/ --delay 3 --batch-size 30 --resume

# Using custom configuration (still supported)
afdd --input stocks.csv --config production_config.yaml --verbose

2. Interactive Mode (For URL Discovery Issues)

# Enable interactive prompts for failed URL discovery (tijori_finance)
afdd --input stocks.csv --website tijori_finance --interactive --verbose

# Interactive with resume (skip existing, prompt for new failures)
afdd --input stocks.csv --website tijori_finance --output ./downloads/ --interactive --resume --verbose

# Interactive with custom settings and website selection
afdd --input stocks.csv --website screener --interactive --delay 5 --verbose

# Cross-website interactive workflow
afdd --input stocks.csv --website tijori_finance --interactive --output ./primary/
afdd --retry-from-report ./primary/afdd_report_*.txt --website screener --interactive --output ./fallback/

3. Report-Based Retry Operations

# Generate retry file from failed downloads report
afdd --retry-from-report ./downloads/afdd_report_20250726_013058.txt

# Custom output location for retry file
afdd --retry-from-report ./reports/failed_report.txt --output ./retry_files/

# After generating retry file, process it with different website
afdd --input ./failed_stocks_retry_20250726_143052.csv --website tijori_finance --interactive --verbose
afdd --input ./failed_stocks_retry_20250726_143052.csv --website screener --interactive --verbose

# Cross-website retry strategy
afdd --retry-from-report ./tijori_downloads/afdd_report_*.txt --website screener --output ./screener_fallback/

4. CSV Sanitization (Fix URL Issues)

# Sanitize CSV with URLs in company name column
afdd --sanitize-csv ./failed_stocks_retry.csv

# After sanitization, process the cleaned file
afdd --input ./failed_stocks_retry_sanitized.csv --interactive --verbose

5. Combined Workflow Examples

# Complete workflow for handling failures:

# Step 1: Initial download attempt
afdd --input stocks.csv --output ./downloads/ --interactive --resume --verbose

# Step 2: If failures occurred, retry from report
afdd --retry-from-report ./downloads/afdd_report_20250726.txt --output ./downloads/

# Step 3: If URLs were accidentally entered, sanitize
afdd --sanitize-csv ./downloads/failed_stocks_retry_20250726.csv

# Step 4: Process sanitized retry file
afdd --input ./downloads/failed_stocks_retry_20250726_sanitized.csv --interactive --verbose

6. Performance Optimization Examples

# Fast processing (minimal delays)
afdd --input stocks.csv --delay 1 --batch-size 100 --resume

# Maximum speed (skip file validation - use with caution!)
afdd --input stocks.csv --delay 1 --batch-size 100 --resume --skip-validation

# Conservative processing (respectful to servers)
afdd --input stocks.csv --delay 5 --batch-size 20 --resume --verbose

# Overnight processing (large datasets)
afdd --input large_dataset.xlsx --output ./overnight_data/ --delay 2 --resume

7. Development & Testing

# Debug mode (non-headless browser)
afdd --input test_stocks.csv --verbose --config debug_config.yaml

# Test with small dataset
afdd --input sample_10_stocks.csv --output ./test/ --verbose

# Resume interrupted test
afdd --input sample_10_stocks.csv --output ./test/ --resume --verbose

Option Combinations Matrix

Use Case Required Options Optional Additions Description
First Run --input stocks.csv --verbose --output ./data/ Initial download attempt
Production --input stocks.csv --resume --verbose --delay 3 Recommended for regular use
Handle Failures --input stocks.csv --interactive --resume --verbose Deal with URL discovery issues
Retry Failed --retry-from-report report.txt --output ./retry/ Process failed stocks manually
Fix URLs --sanitize-csv file.csv N/A Convert URLs to company names
Large Datasets --input large.xlsx --resume --delay 3 --batch-size 30 Optimize for bulk processing
Maximum Speed --input stocks.csv --resume --skip-validation --delay 1 --batch-size 100 Fastest processing (skip file validation)
Sync Directories --sync-directories --verbose --output ./target/ Sync Excel files and consolidate URL caches

Quick Command Reference

# Most common commands
afdd --input stocks.csv --resume --verbose                    # Standard production use
afdd --input stocks.csv --interactive --resume --verbose      # Handle difficult stocks
afdd --retry-from-report report.txt                          # Retry failed downloads
afdd --sanitize-csv retry_file.csv                          # Fix URL formatting issues
afdd --sync-directories --verbose                            # Sync directories and caches
afdd --sync-directories --sync-source ./a --sync-target ./b  # Sync specific directories

Command Line Options

Option Description Default
--input, -i Input file (CSV/Excel) with stock symbols Required*
--output, -o Output directory for downloaded files ./downloads
--website, -w Website to download from tijori_finance
--list-websites List available websites and exit False
--delay, -d Delay between downloads (seconds) 2
--batch-size, -b Stocks per batch 50
--resume Skip already downloaded files False
--skip-validation Skip content validation of existing files (faster but less safe) False
--verbose, -v Enable detailed logging False
--interactive, -I Enable interactive mode for failed URL discovery False
--config, -c Custom configuration file config.yaml
--retry-from-report, -r Retry failed stocks from an AFDD report file None
--sanitize-csv, -s Sanitize CSV by converting URLs to company names None
--sync-directories Sync files between directories and consolidate URL caches False
--sync-source Source directory for sync (overrides config) None
--sync-target Target directory for sync (overrides config) None

*Required except when using --retry-from-report, --sanitize-csv, --sync-directories, or --list-websites

🌐 Website Options

Website Key Full Name Authentication Data Format Rate Limit
tijori_finance Tijori Finance Premium account Excel (5 sheets) 2s delay
tijori Tijori Finance (alias) Premium account Excel (5 sheets) 2s delay
screener Screener.in Optional/Free CSV 3s delay
screener_in Screener.in (alias) Optional/Free CSV 3s delay

Input File Formats

Standard Format (Required: symbol + company name)

symbol,name
RELIANCE,Reliance Industries Limited
TCS,Tata Consultancy Services Limited
HDFCBANK,HDFC Bank Limited

Legacy Format (symbol only - still supported)

symbol,segment,market_cap
RELIANCE,Oil Refining,1911867
TCS,Software Services,1143225
HDFCBANK,Banks,1538758

Supported File Types:

  • .csv (CSV files)
  • .xlsx and .xls (Excel files)

Supported Column Names:

  • Symbols: symbol, ticker, stock, stock_symbol
  • Company Names: name, company_name, company, name_of_company

Note: The 2-column format (symbol + company name) provides better URL discovery accuracy and is recommended for optimal results.

Configuration

AFDD now uses a profile-based configuration system where each website has its own configuration profile. The traditional config.yaml is still supported for global settings.

🌐 Website Profiles (New Approach)

Each website has its own profile in afdd/profiles/:

Tijori Finance Profile (tijori_finance.yaml)

website:
  name: "Tijori Finance"
  base_url: "https://www.tijorifinance.com"

authentication:
  method: "interactive_login"
  required: true

url_templates:
  patterns:
    - template: "https://www.tijorifinance.com/company/${company_slug}/"
      variables:
        company_slug:
          transformations: ["clean_special_chars", "standardize_suffix", "slug"]

selectors:
  download_dropdown:
    selectors:
      - strategy: "xpath"
        value: "//button[@class='dropdown-toggle' and @title='Download financial statements']"

download_settings:
  filename_pattern: "{symbol}_{date}.xlsx"
  preferred_download_type: "consolidated"

Screener.in Profile (screener.yaml)

website:
  name: "Screener.in"
  base_url: "https://www.screener.in"

authentication:
  method: "optional"
  required: false

rate_limiting:
  delay_between_requests: 3
  max_requests_per_minute: 15

download_settings:
  filename_pattern: "{symbol}_screener_{date}.csv"
  preferred_format: "csv"

📁 Traditional Configuration (Still Supported)

Global Settings (config.yaml)

# Global download settings (applied to all websites)
download:
  delay_seconds: 2          # Respectful delay between downloads
  batch_size: 50           # Process in batches
  timeout_seconds: 60      # Download timeout
  max_retries: 3           # Retry failed downloads

# Global browser settings
browser:
  headless: true           # Run in background (set false for debugging)
  window_size: "1920,1080" # Browser window size

# Output settings
output:
  create_summary: true     # Generate reports
  save_failed_list: true

⚙️ Configuration Hierarchy

  1. Website Profile (highest priority)
  2. CLI Arguments (overrides profile)
  3. Global Config (fallback for missing profile settings)
  4. Built-in Defaults (final fallback)
# Global config affects all websites
afdd --input stocks.csv --config global_settings.yaml

# Website profiles automatically loaded
afdd --input stocks.csv --website tijori_finance  # Uses tijori_finance.yaml
afdd --input stocks.csv --website screener        # Uses screener.yaml

# CLI overrides both global and profile settings
afdd --input stocks.csv --website tijori_finance --delay 5 --batch-size 20

🎯 Production Settings

For Large-Scale Processing (200+ stocks)

# Global config.yaml
download:
  delay_seconds: 2         # Be respectful to servers
  batch_size: 50          # Optimal batch size
  
browser:
  headless: true          # Run in background
  
logging:
  level: "INFO"           # Reduce log verbosity

Website-Specific Optimization

# Tijori Finance: Premium quality, moderate speed
afdd --input stocks.csv --website tijori_finance --delay 2 --batch-size 50

# Screener.in: Free service, conservative rate limiting
afdd --input stocks.csv --website screener --delay 3 --batch-size 20

Output Structure

output_directory/
├── RELIANCE_2025-01-25.xlsx    # Downloaded data
├── TCS_2025-01-25.xlsx          # Downloaded data
├── HDFCBANK_2025-01-25.xlsx     # Downloaded data
├── afdd_report_20250125.txt     # Detailed report
├── afdd_20250125.log            # Execution log
└── url_mappings_cache.json      # URL cache

Each Excel file contains 5 sheets:

  • BalanceSheet: Balance sheet data
  • Profit&Loss: P&L statements
  • CashFlow: Cash flow statements
  • Ratios: Financial ratios
  • QuarterlyResults: Quarterly results

Real-World Examples

F&O Stock Processing (Most Common)

# Standard F&O processing (recommended approach)
afdd --input fo_stocks.xlsx --output ./fo_downloads/ --resume --verbose --interactive

# Expected performance: 216 F&O stocks in ~7-10 minutes
# Interactive mode handles the ~10-15 stocks that typically have URL issues

Large Dataset Processing (500+ Stocks)

# Overnight batch processing
afdd --input nse_complete.xlsx --output ./bulk_data/ --delay 3 --batch-size 50 --resume

# If issues occur, use interactive retry workflow
afdd --retry-from-report ./bulk_data/afdd_report_*.txt --output ./bulk_data/
afdd --input ./bulk_data/failed_stocks_retry_*.csv --interactive --verbose

Resume Interrupted Downloads

# Simple resume (most common)
afdd --input stocks.csv --output ./downloads/ --resume --verbose

# Resume with interactive mode for new failures
afdd --input stocks.csv --output ./downloads/ --resume --interactive --verbose

# Fast resume (skip file content validation)
afdd --input stocks.csv --output ./downloads/ --resume --skip-validation --verbose

Skip Validation Mode (Performance Optimization)

⚠️ Use with caution - This option skips content validation of existing files for maximum speed.

# Maximum speed processing (recommended for trusted environments)
afdd --input fo_stocks.xlsx --output ./fo_downloads/ --resume --skip-validation --verbose

# Benefits:
# - 2-3x faster when many files already exist
# - No Excel file parsing for existing files
# - Ideal for re-running on partially complete datasets

# Trade-offs:
# - Corrupted files won't be detected and re-downloaded
# - Missing sheets in Excel files won't be caught
# - Files with 0 bytes will still be skipped (basic check remains)

When to use --skip-validation:

  • ✅ Re-running on datasets where most files already exist
  • ✅ Production environments with reliable storage
  • ✅ Time-critical processing where speed is paramount
  • ❌ First-time downloads (no benefit)
  • ❌ Unreliable storage or network environments
  • ❌ When data integrity is critical

Directory Synchronization

⚙️ Sync files between directories and consolidate URL caches automatically

# Basic directory sync (uses config.yaml settings)
afdd --sync-directories --verbose

# Specify directories via CLI (overrides config)
afdd --sync-directories --sync-source ./downloads --sync-target ./fo_downloads --verbose

# Mix CLI and config (CLI overrides config settings)
afdd --sync-directories --sync-source ./custom_source --verbose

# Sync with custom configuration file
afdd --sync-directories --config custom_sync_config.yaml --verbose

# Benefits:
# - Copies missing Excel files between directories
# - Consolidates URL mapping caches (prioritizes proper company names)
# - Removes duplicate _unknown entries
# - Creates backups before overwriting files

Configuration (config.yaml):

sync:
  source_directory: "./downloads"      # Source for files to copy
  target_directory: "./fo_downloads"   # Target destination
  backup_replaced_files: true          # Backup before overwriting
  consolidate_caches: true             # Merge URL caches

Directory Specification Options:

  1. Config File Only (default):

    afdd --sync-directories --verbose
# Uses config.yaml: sync.source_directory and sync.target_directory

  2. CLI Override (recommended for ad-hoc syncing):

    afdd --sync-directories --sync-source ./my_downloads --sync-target ./archive --verbose
# CLI directories override config settings

  3. Mixed Approach:

    afdd --sync-directories --sync-source ./custom_source --verbose
# Uses CLI source + config target

When to use --sync-directories:

  • ✅ After running downloads in multiple directories
  • ✅ When URL caches have become fragmented
  • ✅ To consolidate data from different download sessions
  • ✅ Before archiving or backing up download results

Advanced Workflow: Complete Failure Recovery

# 1. Initial attempt with interactive prompts
afdd --input stocks.csv --output ./data/ --interactive --resume --verbose

# 2. If report shows many failures, extract them for manual retry
afdd --retry-from-report ./data/afdd_report_20250726_*.txt --output ./data/

# 3. If you accidentally entered URLs instead of company names
afdd --sanitize-csv ./data/failed_stocks_retry_*.csv

# 4. Process the cleaned retry file
afdd --input ./data/failed_stocks_retry_*_sanitized.csv --interactive --verbose

# 5. Sync and consolidate results from multiple directories
afdd --sync-directories --verbose

Production Deployment Examples

# Daily automated runs
afdd --input daily_watchlist.csv --output ./daily_data/ --resume --delay 2

# Weekly bulk updates  
afdd --input weekly_universe.xlsx --output ./weekly_data/ --resume --verbose --delay 3

# Monthly comprehensive download
afdd --input monthly_complete.xlsx --output ./monthly_data/ --resume --interactive --verbose

How It Works

Browser Automation

  1. Opens Chrome browser and navigates to Tijori Finance
  2. Prompts user to log in (interactive login confirmation)
  3. Detects dropdown buttons using intelligent selectors
  4. Clicks download links (prefers "Standalone" over "Consolidated")
  5. Monitors download completion with fast file detection

File Processing

  1. Detects new files using timestamp comparison
  2. Renames files to standardized format (SYMBOL_DATE.xlsx)
  3. Validates downloads by checking file size and content
  4. Handles duplicates by removing old files before renaming

Advanced Features

  • Interactive URL Discovery: User prompts when primary URL construction fails
  • Intelligent Suffix Variations: Tries "Limited" → "Ltd", "Corporation" → "Corp" automatically
  • Browser State Optimization: Avoids duplicate navigation for performance
  • Report-Based Retry: Extract failed stocks from reports for manual company name input
  • CSV Sanitization: Automatically converts URLs to proper company names
  • Enhanced Caching: Improved URL discovery with cache invalidation and retry logic

Error Handling

  • Automatic retries for failed downloads
  • URL discovery fallback with multiple suffix variations
  • Interactive prompts for user-assisted URL discovery
  • Manual review flagging for stocks requiring attention
  • Comprehensive logging for troubleshooting

Development

🔧 Creating Custom Adapters

AFDD's plugin architecture makes it easy to add support for new financial websites.

Step 1: Create Website Profile

Create afdd/profiles/mywebsite.yaml:

website:
  name: "My Financial Website"
  base_url: "https://www.mywebsite.com"
  description: "Custom financial data source"

authentication:
  method: "api_key"  # or "interactive_login", "oauth", "none"
  required: true

url_templates:
  patterns:
    - template: "https://www.mywebsite.com/stocks/${symbol_upper}"
      priority: 100
      variables:
        symbol_upper:
          transformations: ["uppercase"]

selectors:
  download_button:
    selectors:
      - strategy: "css"
        value: ".download-btn"
        description: "Main download button"
      - strategy: "xpath" 
        value: "//button[contains(text(), 'Export')]"
        description: "Fallback export button"

file_validation:
  expected_format: "excel"
  min_file_size: 1000

download_settings:
  filename_pattern: "{symbol}_mysite_{date}.xlsx"

Step 2: Implement Adapter Class

Create afdd/adapters/mywebsite.py:

from .base import WebsiteAdapter, StockData, DownloadResult
from ..core.locators import locator_registry
from ..core.templates import URLTemplateBuilder, template_engine

class MyWebsiteAdapter(WebsiteAdapter):
    @property
    def website_name(self) -> str:
        return self.config.get('website', {}).get('name', 'My Website')
    
    @property  
    def base_url(self) -> str:
        return self.config.get('website', {}).get('base_url')
    
    def authenticate(self) -> bool:
        # Implement your authentication logic
        auth_method = self.config.get('authentication', {}).get('method')
        if auth_method == 'api_key':
            return self._api_key_auth()
        elif auth_method == 'interactive_login':
            return self._interactive_login()
        return True
    
    def discover_url(self, stock_data: StockData) -> Optional[str]:
        # Use the template engine for URL construction
        template_vars = {
            'symbol': stock_data.symbol,
            'symbol_upper': stock_data.symbol,
            'company_name': stock_data.company_name or ''
        }
        
        candidate_urls = self.url_builder.build_urls(template_vars)
        
        for url in candidate_urls:
            if self._validate_url(url):
                return url
        return None
    
    def download_data(self, stock_data: StockData, output_dir: Path) -> DownloadResult:
        # Implement download logic using selectors
        url = self.discover_url(stock_data)
        if not url:
            return DownloadResult(False, error_message="URL discovery failed")
        
        # Navigate and download
        self.driver.get(url)
        
        # Use registered selectors
        download_btn = locator_registry.get_locator('mywebsite', 'download_button')
        if download_btn and download_btn.find_element(self.driver):
            # Implement download logic
            pass
        
        return DownloadResult(True, filename="downloaded_file.xlsx")
    
    def validate_download(self, file_path: Path, stock_data: StockData) -> Tuple[bool, str]:
        # Implement file validation
        return True, "File is valid"
    
    def get_expected_filename(self, stock_data: StockData) -> str:
        pattern = self.config.get('download_settings', {}).get('filename_pattern')
        date_str = datetime.now().strftime('%Y-%m-%d')
        return pattern.format(symbol=stock_data.symbol, date=date_str)

Step 3: Register Adapter

Add to afdd/adapters/__init__.py:

from .mywebsite import MyWebsiteAdapter

ADAPTER_REGISTRY['mywebsite'] = MyWebsiteAdapter

Step 4: Use Your Custom Adapter

# List websites (should now include your adapter)
afdd --list-websites

# Use your custom adapter
afdd --input stocks.csv --website mywebsite --output ./custom_data/

🛠️ Template System Usage

URL Templates with Transformations

url_templates:
  patterns:
    - template: "https://example.com/company/${clean_name}/"
      variables:
        clean_name:
          transformations:
            - "remove_suffix"       # Remove "Limited", "Corp", etc.
            - "clean_special_chars" # Remove &, @, etc.  
            - "slug"               # Convert to URL format

Available Transformations

  • lowercase / uppercase - Case conversion
  • slug - URL-friendly format
  • remove_suffix - Remove corporate suffixes
  • standardize_suffix - Normalize suffix formats
  • clean_special_chars - Remove special characters
  • hyphenate / dehyphenate - Space/hyphen conversion

🎯 Selector Configuration

Fallback Selector Chains

selectors:
  my_button:
    selectors:
      - strategy: "css"
        value: ".primary-btn"
        timeout: 10
      - strategy: "xpath"
        value: "//button[contains(text(), 'Download')]"
        timeout: 5
      - strategy: "css"
        value: ".fallback-btn" 
        timeout: 3
    required: true
    retry_count: 3

🧪 Testing & Development

# System health check
python -c "from afdd.utils import system_health_check; system_health_check()"

# Validate input file
python -c "from afdd.utils import validate_input_file; from pathlib import Path; validate_input_file(Path('stocks.csv'))"

# Test adapter creation
python -c "from afdd.adapters import AdapterFactory; print(AdapterFactory.list_available_adapters())"

# Test profile loading
python -c "from afdd.adapters import AdapterFactory; print(AdapterFactory.load_profile('mywebsite'))"

📋 Code Quality

# Format code
black afdd/

# Lint code
flake8 afdd/

# Test new adapters
afdd --input sample_stocks.csv --website mywebsite --output ./test/ --verbose

Troubleshooting

Common Issues & Solutions

"ChromeDriver not found"

# Automatic installation (with conda environment)
python -c "from webdriver_manager.chrome import ChromeDriverManager; ChromeDriverManager().install()"

"Not logged in" warnings

# Solution: Make sure you're logged into Tijori Finance
# 1. Browser will open automatically
# 2. Log in to your premium account  
# 3. Type 'logged in' when prompted

"URL discovery failed" (Legacy Issue - Now Mostly Solved)

# OLD APPROACH: Manual review required (~5-10% of stocks)
# NEW APPROACH: Use interactive mode and report-based retry

# Step 1: Run with interactive mode
afdd --input stocks.csv --interactive --resume --verbose

# Step 2: For remaining failures, use report-based retry
afdd --retry-from-report ./downloads/afdd_report_*.txt

"CSV contains URLs instead of company names"

# If you accidentally entered URLs during retry process
afdd --sanitize-csv ./failed_stocks_retry.csv

# Then process the sanitized file
afdd --input ./failed_stocks_retry_sanitized.csv --interactive --verbose

"Too many interactive prompts"

# Use report-based retry to handle bulk failures efficiently
afdd --retry-from-report ./downloads/afdd_report_*.txt --output ./downloads/

# This allows you to batch-input company names instead of one-by-one prompts

"Slow downloads (60+ seconds per stock)"

# This shouldn't happen with the optimized version
# If it does, check your internet connection
# Consider increasing timeout in config.yaml

🌐 Multi-Website Troubleshooting

Website Selection Issues

# Check available websites
afdd --list-websites

# Verify website key is correct
afdd --input stocks.csv --website tijori_finance    # Correct
afdd --input stocks.csv --website tijori           # Also correct (alias)
afdd --input stocks.csv --website invalid_name     # Will show error

# Check if profile exists
python -c "from afdd.adapters import AdapterFactory; print(AdapterFactory.load_profile('screener'))"

Profile Loading Errors

# Check if profile file exists
ls -la afdd/profiles/tijori_finance.yaml
ls -la afdd/profiles/screener.yaml

# Validate YAML syntax
python -c "import yaml; yaml.safe_load(open('afdd/profiles/tijori_finance.yaml'))"

# Debug profile loading
afdd --input stocks.csv --website tijori_finance --verbose

Website-Specific Issues

Tijori Finance Problems

# Authentication issues
afdd --input stocks.csv --website tijori_finance --verbose
# Make sure you're logged into premium account

# Premium account required
# Free accounts will not work with Tijori Finance adapter

Screener.in Problems

# Rate limiting issues
afdd --input stocks.csv --website screener --delay 5 --batch-size 10 --verbose

# Free service limitations
# Some data may not be available without subscription

Cross-Website Issues

# Different file formats
afdd --input stocks.csv --website tijori_finance    # Creates .xlsx files
afdd --input stocks.csv --website screener          # Creates .csv files

# Website-specific failures
afdd --retry-from-report ./tijori_downloads/afdd_report_*.txt --website screener

🛠️ Adapter Development Troubleshooting

Custom Adapter Not Found

# Check registration
python -c "from afdd.adapters import ADAPTER_REGISTRY; print(ADAPTER_REGISTRY.keys())"

# Verify import works
python -c "from afdd.adapters.mywebsite import MyWebsiteAdapter"

Profile Loading Issues

# Check YAML syntax
python -c "import yaml; yaml.safe_load(open('afdd/profiles/mywebsite.yaml'))"

# Test profile loading
python -c "from afdd.adapters import AdapterFactory; print(AdapterFactory.load_profile('mywebsite'))"

Template Issues

# Test URL template rendering
python -c "
from afdd.core.templates import URLTemplateBuilder
builder = URLTemplateBuilder()
print(builder.build_urls({'symbol': 'RELIANCE'}))
"

🔧 Legacy Feature Troubleshooting

Interactive Mode Not Working

# Make sure you're using the -I or --interactive flag
afdd --input stocks.csv --website tijori_finance --interactive --verbose

# Check that your terminal supports interactive input
# Some IDEs may not support builtins.input() properly

Report Parsing Errors

# Ensure report file exists and is not corrupted
ls -la ./downloads/afdd_report_*.txt

# If report is truncated, re-run the original download
afdd --input stocks.csv --website tijori_finance --resume --verbose

Sanitization Issues

# Check CSV format - must have 'symbol' and 'name' columns
head -5 ./failed_stocks_retry.csv

# Verify URLs are in the expected format
# Should be: https://www.tijorifinance.com/company/slug/

Getting Help

  1. Check logs: Look for .log files in output directory
  2. Review reports: Check afdd_report_*.txt for detailed information
  3. Health check: Run system health check to validate setup
  4. GitHub Issues: Report bugs with log files attached

Performance

🌐 Multi-Website Performance

Speed Metrics by Website

Website Typical Speed Authentication Success Rate Best Use Case
Tijori Finance 2-3s per stock Interactive login ~95% (interactive) Premium comprehensive data
Screener.in 3-4s per stock Optional ~85% Free screening data

Comparative Performance

# Tijori Finance: Premium quality, moderate speed
afdd --input stocks.csv --website tijori_finance --delay 2 --batch-size 50
# Expected: 216 F&O stocks in ~7-10 minutes

# Screener.in: Free service, conservative speed  
afdd --input stocks.csv --website screener --delay 3 --batch-size 20
# Expected: 216 stocks in ~12-15 minutes

🚀 Multi-Website Optimization Strategies

Strategy 1: Parallel Multi-Website Processing

# Run different websites simultaneously in different terminals
# Terminal 1:
afdd --input stocks.csv --website tijori_finance --output ./tijori/ --verbose

# Terminal 2: 
afdd --input stocks.csv --website screener --output ./screener/ --verbose

# Fastest way to get data from multiple sources

Strategy 2: Fallback Performance

# Primary website with fallback (optimized workflow)
afdd --input stocks.csv --website tijori_finance --interactive --output ./primary/
afdd --retry-from-report ./primary/afdd_report_*.txt --website screener --output ./fallback/

# Gets premium data where available, free data for failures

Strategy 3: Cost-Optimized Performance

# Start with free data (faster processing)
afdd --input stocks.csv --website screener --delay 2 --batch-size 25 --output ./free/

# Then get premium data for specific high-priority stocks only
afdd --input priority_stocks.csv --website tijori_finance --output ./premium/

⚡ Website-Specific Optimization

Tijori Finance Optimization

# Fastest Tijori settings (premium account)
afdd --input stocks.csv --website tijori_finance --delay 1 --batch-size 100 --resume --skip-validation

# Balanced Tijori (recommended)
afdd --input stocks.csv --website tijori_finance --delay 2 --batch-size 50 --interactive --resume --verbose

# Conservative Tijori (maximum reliability)
afdd --input stocks.csv --website tijori_finance --delay 3 --batch-size 20 --interactive --resume --verbose

Screener.in Optimization

# Respectful free service usage (recommended)
afdd --input stocks.csv --website screener --delay 3 --batch-size 20 --resume --verbose

# Faster processing (use with caution on free service)
afdd --input stocks.csv --website screener --delay 2 --batch-size 30 --resume

# Conservative (maximum server respect)
afdd --input stocks.csv --website screener --delay 5 --batch-size 10 --verbose

📊 Success Rates & Reliability

By Website Type

  • Tijori Finance:

    • Standard Mode: ~90% success rate
    • Interactive Mode: ~95-98% success rate
    • Template-based URL discovery with manual fallback

  • Screener.in:

    • Standard Mode: ~85% success rate
    • Interactive Mode: ~90% success rate
    • Simpler page structure, fewer edge cases

Cross-Website Success

  • Multi-Website Fallback: ~98% combined success rate
  • Template Engine: Handles complex name variations automatically
  • Report-Based Retry: 100% success rate for manually input company names

🎯 Performance Recommendations

For Production Use (200+ stocks)

# Option 1: Single premium source (fastest)
afdd --input stocks.csv --website tijori_finance --delay 2 --batch-size 50 --resume --interactive

# Option 2: Multi-source strategy (most comprehensive)  
afdd --input stocks.csv --website tijori_finance --output ./primary/ --resume --interactive
afdd --retry-from-report ./primary/afdd_report_*.txt --website screener --output ./fallback/

# Option 3: Cost-optimized (free first, premium for failures)
afdd --input stocks.csv --website screener --output ./free/ --resume
afdd --retry-from-report ./free/afdd_report_*.txt --website tijori_finance --output ./premium/

For Development/Testing

# Quick test with small dataset
afdd --input sample_10_stocks.csv --website screener --verbose

# Cross-website compatibility test
afdd --input sample_stocks.csv --website tijori_finance --output ./test_tijori/
afdd --input sample_stocks.csv --website screener --output ./test_screener/

🔧 Performance Monitoring

# Monitor website availability
afdd --list-websites

# Test specific website performance
afdd --input single_stock.csv --website tijori_finance --verbose
afdd --input single_stock.csv --website screener --verbose

# Compare performance across websites
time afdd --input sample_stocks.csv --website tijori_finance --output ./perf_test_1/
time afdd --input sample_stocks.csv --website screener --output ./perf_test_2/

License

MIT License - see LICENSE file for details.

Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Support

  • GitHub Issues: Report bugs or request features
  • Documentation: This README, inline code documentation, and examples/usage_examples.md
  • System Health: Use built-in health check commands
  • Multi-Website Help: Use afdd --list-websites for available adapters

🆕 Migration from Single-Website Version

If you're upgrading from an older AFDD version:

# Old usage (still works - defaults to tijori_finance)
afdd --input stocks.csv --output ./downloads/ --resume --verbose

# New explicit usage (recommended)
afdd --input stocks.csv --website tijori_finance --output ./downloads/ --resume --verbose

# Explore new capabilities
afdd --list-websites
afdd --input stocks.csv --website screener --output ./downloads/ --verbose

Backward Compatibility: All existing commands and configurations continue to work unchanged.

Built with ❤️ for the financial analysis community

Transforming from single-purpose tool to multi-website financial data framework

🌐 Multi-Website Ready • 🔧 Extensible Architecture • 📊 Production Tested • 🚀 Easy to Use