Skip to content

Updating request inside "default_handler", "error_handler" and "failed_request_handler" #1483

@ericvg97

Description

@ericvg97

Is there a way to update the request.user_data inside these handlers? I would like to store info about the last error, or the timestamp of the different retries.

This is my handler

async def default_handler(context: AdaptivePlaywrightCrawlingContext) -> None:
        context.log.info(f"Processing {context.request.url} ...")
        context.request.user_data["something"] = "something"
        await context.enqueue_links(strategy="same-domain")

crawler.router.default_handler(default_handler)

but looking at the db this field is not being saved. I just see ""user_data":{"__crawlee":{"state":3}"

I guess the context is either not saved after that or it is sent as a copy, but not sure from looking at the code. Any other way of doing this that I am not seeing?

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions