transfermarkt-scraper

A web scraper for collecting data from Transfermarkt website. It recurses into the Transfermarkt hierarchy to find competitions, games, clubs, players and appearances, and extract them as JSON objects.

====> Confederations ====> Competitions ====> (Clubs, Games) ====> Players ====> Appearances

Each one of these entities can be discovered and refreshed separately by invoking the corresponding crawler.

Installation

This project uses Crawlee for Python and can be run with the CLI entry point. All dependencies can be installed using poetry.

cd transfermarkt-scraper
poetry install
poetry shell

Usage

These are some usage examples for how the scraper may be run.

# discover confederations and competitions on separate invocations
python -m tfmkt confederations > confederations.json
python -m tfmkt competitions -p confederations.json > competitions.json

# you can use intermediate files or pipe crawlers one after the other to traverse the hierarchy
cat competitions.json | head -2 \
    | python -m tfmkt clubs \
    | python -m tfmkt players \
    | python -m tfmkt appearances

Alternatively you can also use dcaribou/transfermarkt-scraper docker image

docker run \
    -ti -v "$(pwd)"/.:/app \
    dcaribou/transfermarkt-scraper:main \
    python -m tfmkt competitions -p samples/confederations.json

Items are extracted in JSON format with one JSON object per item (confederation, league, club, player or appearance), which get printed to the stdout. Samples of extracted data are provided in the samples folder.

Check out transfermarkt-datasets to see transfermarkt-scraper in action on a real project.

arguments

• -p / --parents: Crawler "parents" are either a file or a piped output with the parent entities. For example, competitions is parent of clubs, which in turn is a parent of players.
• -s / --season: The season that the crawler is to run for. It defaults to the most recent season.
• --base-url: Override the base Transfermarkt URL.

contribute

Extending existing crawlers in this project in order to scrape additional data or even creating new crawlers is quite straightforward. If you want to contribute with an enhancement to transfermarkt-scraper I suggest that you follow a workflow similar to

1. Fork the repository
2. Modify or add new crawlers to tfmkt/crawlers. Here is an example PR that extends the games crawler to scrape a few additional fields from Transfermakt games page.
3. Create a PR with your changes and a short description for the enhancement and send it over 🚀

It is usually also a good idea to have a short discussion about the enhancement beforehand. If you want to propose a change and collect some feeback before you start coding you can do so by creating an issue with your idea in the Issues section.