Implement/document a way how to pass information between handlers

[honzajavorek]

I came across a situation where I scrape half of the item's data in the listing page handler and the other half in a handler taking care of the detail page. I think must be quite common case. I struggle to see how I pass the information down from one handler to another. See concrete example below:

import re
import asyncio
from enum import StrEnum, auto

import click
from crawlee.beautifulsoup_crawler import (
    BeautifulSoupCrawler,
    BeautifulSoupCrawlingContext,
)
from crawlee.router import Router


LENGTH_RE = re.compile(r"(\d+)\s+min")


class Label(StrEnum):
    DETAIL = auto()


router = Router[BeautifulSoupCrawlingContext]()


@click.command()
def edison():
    asyncio.run(scrape())


async def scrape():
    crawler = BeautifulSoupCrawler(request_handler=router)
    await crawler.run(["https://edisonfilmhub.cz/program"])
    await crawler.export_data("edison.json", dataset_name="edison")


@router.default_handler
async def detault_handler(context: BeautifulSoupCrawlingContext):
    await context.enqueue_links(selector=".program_table .name a", label=Label.DETAIL)


@router.handler(Label.DETAIL)
async def detail_handler(context: BeautifulSoupCrawlingContext):
    context.log.info(f"Scraping {context.request.url}")

    description = context.soup.select_one(".filmy_page .desc3").text
    length_min = LENGTH_RE.search(description).group(1)
    # TODO get starts_at, then calculate ends_at

    await context.push_data(
        {
            "url": context.request.url,
            "title": context.soup.select_one(".filmy_page h1").text.strip(),
            "csfd_url": context.soup.select_one(".filmy_page .hrefs a")["href"],
        },
        dataset_name="edison",
    )

I need to scrape starts_at at the default_handler, then add more details to the item on the detail page, and calculate the ends_at time according to the length of the film. Even if I changed enqueue_links to something more delicate, how do I pass data from one request to another?

[vdusek]

You probably want to use the user_data argument of enqueue_links:

@crawler.router.default_handler
async def default_handler(context: BeautifulSoupCrawlingContext) -> None:
    context.log.info(f'Processing {context.request.url}')

    extracted_data_in_default_handler = context.soup.title.string

    await context.enqueue_links(
        user_data={'extracted_data_in_default_handler': extracted_data_in_default_handler},
    )

[honzajavorek]

That sounds about right. Haven't found much about it in the docs, at least using the built-in search. How do I access it in the other handler? Something like context.user_data?

[B4nan]

It's an attribute of the request, so you should be able to use context.request.user_data.

[honzajavorek]

Cool, thanks! This was my main blocker when developing kino over the weekend. Can't promise getting back to this soon as it's just a hobby thing, but I'll assume this is enough info for me to make it work. Feel free to close this unless you want to turn it into a tracking issue of "this needs more examples in the docs".

[vdusek]

Cool, thanks! This was my main blocker when developing kino over the weekend. Can't promise getting back to this soon as it's just a hobby thing, but I'll assume this is enough info for me to make it work. Feel free to close this unless you want to turn it into a tracking issue of "this needs more examples in the docs".

Great, let us know once you try it. IMO we should add some examples to docs regarding this topic, so we can leave this open.

[honzajavorek]

It works, but it has a surprising quirk. It stringifies all dict values. It's not JSON, it's just as if I casted str() around the value. And it's not applied to the data I send, but to the values of a dict I send. This must be a bug. Minimal example:

@router.default_handler
async def detault_handler(context: BeautifulSoupCrawlingContext):
    timetable = defaultdict(set)
    for i in range(5):
        url = f"https://example.com/{i}"
        timetable[url].add(datetime(2024, 11, 2, 15, 30), tzinfo=zoneinfo.ZoneInfo(key='Europe/Prague'))
    await context.enqueue_links(
        selector=".program_table .name a",
        user_data={"timetable": timetable},
        label="detail",
    )

@router.handler("detail")
async def detail_handler(context: BeautifulSoupCrawlingContext):
    for url, starts_ats in context.request.user_data["timetable"].items():
        print(starts_ats)  # {datetime.datetime(2024, 11, 2, 15, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Prague'))}
        print(type(starts_ats))  # <class 'str'>

I didn't play with it further, so I don't know what else it stringifies this way, but obviously such data is unusable. If this is expected behavior, I'd have to JSON-encode those and then JSON-decode. I could use list instead of set, that's okay, but I'd have to manually .isoformat() all the dates and then parse them to get back native types. That feels like a lot of unnecessary boilerplate which shouldn't be needed or if it's needed because of certain architecture of the framework, it should happen automatically.

[honzajavorek]

As a separate comment I want to also add that I noticed there is nothing more delicate than enqueue_links(). What I happen to need is to traverse a timetable for times of movie screenings and then scrape individual movie URLs for details about the movie. What would be most natural to me is something like:

# JUST PSEUDOCODE
for item in soup.select(".item"):
    time = datetime.fromisoformat(item.select_one(".screening-time").text)
    link = item.select_one(".movie-link")
    context.enqueue_link(link, user_data={"time": time}, label="detail")

This is an approach I'm used to do when creating scrapers, all my life. With both one-time scripts without any framework or with Scrapy. I don't believe I'm alone.

As far as I understand, there's currently no way to do that in Crawlee. I'm forced to scrape the whole timetable and then pass it down to the detail handler. In the detail handler, I (want to, if not for the bug above) look up the movie in the timetable by its URL and get the screening times. This feels unnatural.

I'm not sure if I do it wrong all the time and the way Crawlee does it is the preferred way to do this for some good reasons, or whether it's just bad UX (actually DX).

Because... Getting the timetable first and then pairing back the movies takes care of duplicate requests and forces me to think about the situation when the same movie is screened multiple times. Crawlee forced me into unnatural architecture of my scraper, resulting in a better algorithm!

Should I ask for .enqueue_link() in a feature request or should I admit that not having it is smart? I don't know. Maybe there are other cases, when enqueuing link by link with data attached would totally make sense. What do you think?

[janbuchar]

As a separate comment I want to also add that I noticed there is nothing more delicate than enqueue_links(). What I happen to need is to traverse a timetable for times of movie screenings and then scrape individual movie URLs for details about the movie. What would be most natural to me is something like:

I'm not sure I understand the whole thought process, but the example you posted is not far from being correct. You could just do this:

for item in soup.select(".item"):
    time = datetime.fromisoformat(item.select_one(".screening-time").text)
    link = item.select_one(".movie-link")
    context.add_requests([Request.from_url(link, user_data={"time": time}, label="detail")])

Or even better, you could gather the links in a list and then pass a list of Request objects to a single context.add_requests call.

Did I understand the question right?

[honzajavorek]

I think you did! I completely overlooked the existence of add_requests() and add_requests() 🤯 And being aware about Request.from_url() will be also quite handy 😄 Thank you! ❤️ I'm sorry if this is documented in some tutorials which I might have skipped 🙏 Awesome. So the only issue is that user_data stringifies parts of the data.

[janbuchar]

I think you did! I completely overlooked the existence of add_requests() and add_requests() 🤯 And being aware about Request.from_url() will be also quite handy 😄 Thank you! ❤️ I'm sorry if this is documented in some tutorials which I might have skipped 🙏 Awesome. So the only issue is that user_data stringifies parts of the data.

This is the relevant documentation - https://crawlee.dev/python/docs/guides/request-storage#request-related-helpers. Feel free to suggest a better place, or if you think that there's a place where a link to this page would be useful.

[janbuchar]

It works, but it has a surprising quirk. It stringifies all dict values. It's not JSON, it's just as if I casted str() around the value. And it's not applied to the data I send, but to the values of a dict I send. This must be a bug.

Short answer is, you need to put JSON-serializable data into user_data. I should be able to validate this so that you get an exception instead of nonsensical output.

[honzajavorek]

Feel free to suggest a better place, or if you think that there's a place where a link to this page would be useful.

I think that guide goes through it well, I had to overlook that one. I went through the intro and I only noticed enqueue_links(). Then I tried suggestions in my editor, but I started typing enqueue, looking for methods like enqueue_link(). I think it's just bad luck and nothing particularly wrong on the docs side.

Short answer is, you need to put JSON-serializable data into user_data. I should be able to validate this so that you get an exception instead of nonsensical output.

Uff, so I'll need to do the heavy lifting. Getting exception is definitely better than just being surprised by the result, but sending just JSON-serializable data is very limiting. Especially when working with dates, or, e.g. money (= decimals). I can use Pydantic or something to help me with serialization and deserialization, but it feels strange that I have to do it just to pass a dict from one function to another.

[honzajavorek]

My workaround for now:

from pydantic import RootModel

TimeTable = RootModel[dict[str, set[datetime]]]

@router.default_handler
async def detault_handler(context: BeautifulSoupCrawlingContext):
    ...
    timetable_json = TimeTable(timetable).model_dump_json()
    await context.enqueue_links(
        selector=".program_table .name a",
        user_data=dict(timetable=timetable_json),
        label="detail",
    )

@router.handler("detail")
async def detail_handler(context: BeautifulSoupCrawlingContext):
    ...
    timetable_json = context.request.user_data["timetable"]
    timetable = TimeTable.model_validate_json(timetable_json).model_dump()
    
    for starts_at in timetable[context.request.url]:
        ends_at = starts_at + timedelta(minutes=length_min)
        await context.push_data(...)

Couldn't come up with anything more beautiful.