Skip to content
BranchesTags

Repository files navigation

LinkedIn Job Scraper

Python FastAPI Playwright License

An automated LinkedIn job scraper that collects Data Scientist positions in Israel every 12-24 hours, stores them in a CSV file with historical snapshots, and provides a modern web dashboard for viewing and downloading the data.

Dashboard Screenshot

⚠️ Disclaimer

This project is for educational purposes only. Scraping LinkedIn may violate their Terms of Service. Use at your own risk and responsibility. The authors are not responsible for any account restrictions or legal issues that may arise from using this tool.

🚀 Features

  • Automated Scraping: Playwright-based scraper with anti-detection measures
  • Scheduled Execution: APScheduler runs scraping every 12 hours (configurable)
  • Data Storage: CSV with deduplication and historical timestamps
  • Modern Dashboard: Real-time countdown timer, job table, search, and download
  • REST API: JSON endpoints for data access and scraper control
  • Anti-Detection: Random delays, user-agent rotation, human-like behavior
  • Production-Ready: Logging, error handling, graceful retries

📋 Table of Contents

  1. Requirements
  2. Installation
  3. Configuration
  4. Running the Application
  5. Using the Dashboard
  6. API Reference
  7. Project Structure
  8. How It Works
  9. Limitations
  10. Troubleshooting

📦 Requirements

  • Python 3.10+
  • pip (Python package manager)
  • Chromium browser (installed automatically by Playwright)

🔧 Installation

1. Clone the Repository

git clone https://github.com/yourusername/linkedin-job-scraper.git
cd linkedin-job-scraper

2. Create Virtual Environment

# Windows
python -m venv venv
venv\Scripts\activate

# macOS/Linux
python3 -m venv venv
source venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

4. Install Playwright Browsers

playwright install chromium

5. Create Configuration File

# Copy example configuration
cp .env.example .env

# Edit with your settings (optional)
# nano .env  # or use any text editor

⚙️ Configuration

Edit the .env file to customize the scraper:

# Application Settings
APP_NAME=LinkedIn Job Scraper
DEBUG=false

# Server Settings
HOST=0.0.0.0
PORT=8000

# Scraper Settings
SEARCH_KEYWORDS=Data Scientist
SEARCH_LOCATION=Israel
MAX_JOBS_PER_RUN=50
SCRAPER_HEADLESS=true

# Scheduler Settings
SCRAPE_INTERVAL_HOURS=12
SCHEDULER_ENABLED=true

# Logging
LOG_LEVEL=INFO

Key Configuration Options

Variable Description Default
SEARCH_KEYWORDS Job title to search Data Scientist
SEARCH_LOCATION Location to filter Israel
MAX_JOBS_PER_RUN Maximum jobs per scrape 50
SCRAPE_INTERVAL_HOURS Hours between scrapes 12
SCRAPER_HEADLESS Run browser without UI true

🏃 Running the Application

Option 1: Using the Run Script

python run.py

Option 2: Using Uvicorn Directly

uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

Option 3: Production Deployment

# Install production server
pip install gunicorn

# Run with multiple workers
gunicorn app.main:app -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:8000

Accessing the Dashboard

Open your browser and navigate to:

🖥️ Using the Dashboard

Main Features

  1. Countdown Timer: Shows time until next scheduled scrape
  2. Statistics: Total jobs, unique companies, locations
  3. Job Table: Searchable, sortable list of all scraped jobs
  4. Download CSV: Export all data to a CSV file
  5. Manual Scrape: Trigger an immediate scraping run

Search and Filter

Use the search box to filter jobs by:

  • Job title
  • Company name
  • Location
  • Required degree
  • Experience level

📡 API Reference

GET /api/data

Returns all job listings as JSON.

Query Parameters:

  • limit (int): Maximum jobs to return
  • offset (int): Number of jobs to skip
  • keyword (string): Filter by keyword in title
  • company (string): Filter by company name
  • location (string): Filter by location

Example:

curl "http://localhost:8000/api/data?limit=10&keyword=senior"

GET /api/download

Download the CSV file.

curl "http://localhost:8000/api/download" -o jobs.csv

GET /api/next-run

Get scheduler status and next run time.

Response:

{
  "is_running": true,
  "enabled": true,
  "interval_hours": 12,
  "next_run": "2024-01-15T18:00:00",
  "seconds_until_next_run": 43200,
  "last_run": "2024-01-15T06:00:00",
  "last_run_success": true,
  "jobs_scraped_last_run": 45
}

POST /api/scrape

Trigger a manual scrape.

curl -X POST "http://localhost:8000/api/scrape"

GET /api/stats

Get job statistics.

GET /api/health

Health check endpoint.

📁 Project Structure

linkedin-job-scraper/
├── app/
│   ├── __init__.py            # Package init
│   ├── main.py                # FastAPI application
│   ├── config.py              # Configuration management
│   ├── api/
│   │   ├── __init__.py
│   │   └── routes.py          # API endpoints
│   ├── scraper/
│   │   ├── __init__.py
│   │   ├── linkedin_scraper.py # Playwright scraper
│   │   └── anti_detection.py   # Anti-bot utilities
│   ├── scheduler/
│   │   ├── __init__.py
│   │   └── jobs.py            # APScheduler configuration
│   ├── storage/
│   │   ├── __init__.py
│   │   └── csv_handler.py     # CSV read/write operations
│   ├── utils/
│   │   ├── __init__.py
│   │   └── helpers.py         # Utility functions
│   ├── templates/
│   │   └── index.html         # Dashboard UI
│   └── static/                # Static files (CSS, JS, images)
├── data/
│   └── jobs/                  # CSV output directory
│       └── linkedin_jobs.csv
├── logs/                      # Application logs
├── tests/
│   ├── __init__.py
│   ├── test_scraper.py
│   └── test_api.py
├── .env.example               # Example configuration
├── .gitignore
├── requirements.txt           # Python dependencies
├── run.py                     # Application entry point
└── README.md                  # This file

🔍 How It Works

1. Scraping Process

┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
│   Scheduler      │────▶│   Playwright     │────▶│   CSV Handler    │
│   (12h interval) │     │   Browser        │     │   (deduplicate)  │
└──────────────────┘     └──────────────────┘     └──────────────────┘
         │                       │                        │
         ▼                       ▼                        ▼
   APScheduler            LinkedIn Jobs             CSV File with
   AsyncIOScheduler       Search Page               timestamps

2. Anti-Detection Measures

  • Random Delays: 2-5 seconds between actions
  • User-Agent Rotation: Pool of realistic browser agents
  • Human-like Behavior: Mouse movements, scrolling patterns
  • Browser Fingerprinting: Realistic viewport, timezone, locale
  • Stealth Scripts: Override automation detection properties

3. Data Flow

  1. Scheduler triggers scraping job (or manual trigger via API)
  2. Playwright launches headless Chromium
  3. Scraper navigates to LinkedIn job search
  4. Job cards are extracted with anti-detection delays
  5. Optional: Navigate to each job for detailed info
  6. Results are deduplicated and saved to CSV
  7. Dashboard displays updated data

⚠️ Limitations

LinkedIn Scraping Challenges

  1. Rate Limiting: LinkedIn may temporarily block excessive requests
  2. Login Walls: Some job details require authentication
  3. CAPTCHA: Automated detection may trigger challenges
  4. Dynamic Content: Page structure may change without notice
  5. Legal Risks: Violates LinkedIn Terms of Service

Recommended Practices

  • Use moderate scraping intervals (12-24 hours)
  • Run in headless=false mode for debugging
  • Monitor logs for access denied errors
  • Consider using a residential proxy for better success
  • Keep MAX_JOBS_PER_RUN reasonable (25-50)

🔧 Troubleshooting

Common Issues

"Playwright not found"

pip install playwright
playwright install chromium

"No jobs found"

  • LinkedIn may have changed their page structure
  • Check if the search URL works manually
  • Review logs for detailed error messages

"Access denied" or "Challenge page"

  • Reduce scraping frequency
  • Try running with SCRAPER_HEADLESS=false
  • Consider using a VPN or proxy

"CSV file not created"

# Ensure data directory exists
mkdir -p data/jobs

# Check write permissions
ls -la data/jobs/

Debug Mode

Enable debug mode for verbose logging:

DEBUG=true
LOG_LEVEL=DEBUG
SCRAPER_HEADLESS=false

Logs

Check application logs:

tail -f logs/app.log

🧪 Testing

Run the test suite:

# All tests
pytest

# With coverage
pytest --cov=app

# Specific test file
pytest tests/test_api.py

🤝 Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📧 Contact

For questions or support, please open an issue on GitHub.

Built with ❤️ using FastAPI, Playwright, and APScheduler

About

No description, website, or topics provided.

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages