page_timout does not work for crawler.arun_many

[KanishkNavale]

The code:

async def get_documents_from_urls(urls: List[str]) -> Optional[List[Document]]:
    logger.info("Querying the internet for relevant information...")

    crawler = AsyncWebCrawler()
    await crawler.start()

    crawler_config = CrawlerRunConfig(
        excluded_tags=["form", "nav", "img", "script", "aside", "footer"],
        remove_overlay_elements=True,
        remove_forms=True,
        verbose=False,
        page_timeout=6000,
    )

    results = await crawler.arun_many(
        urls=urls,
        config=crawler_config,
    )

    await crawler.close()
    return generate_docs_from_crawls(results)

The Error:

 × Unexpected error in _crawl_web at line 1205 in _crawl_web (.venv/lib/python3.12/site-                               │
│ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
│   Error: Failed on navigating ACS-GOTO:                                                                               │
│   Page.goto: Timeout 60000ms exceeded.                                                                                │
│   Call log:                                                                                                           │
│   - navigating to "https://tintin.fandom.com/wiki/Tintin", waiting until "domcontentloaded"                           │
│                                                                                                                       │
│                                                                                                                       │
│   Code context:                                                                                                       │
│   1200                                                                                                                │
│   1201                       response = await page.goto(                                                              │
│   1202                           url, wait_until=config.wait_until, timeout=config.page_timeout                       │
│   1203                       )                                                                                        │
│   1204                   except Error as e:                                                                           │
│   1205 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                          │
│   1206                                                                                                                │
│   1207                   await self.execute_hook("after_goto", page, context=context, url=url, response=response)     │
│   1208                                                                                                                │
│   1209                   if response is None:                                                                         │
│   1210                       status_code = 200 

Note:
This works fine for the method crawler.arun()

[aravindkarnam]

As mentioned in #455 , fix for this is ready. Getting released soon.

[LEVIII007]

2025-04-09 13:15:45,962 - INFO - Processing URL: https://www.cabledatasheet.com/korea-south-(129)/ (Name: Korea, South (129))
[ERROR]... × https://www.trademo.com/south-korea/manufacturers/... | Error:
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ × Unexpected error in _crawl_web at line 582 in _crawl_web (.venv/lib/python3.13/site- │
│ packages/crawl4ai/async_crawler_strategy.py): │
│ Error: Failed on navigating ACS-GOTO: │
│ Page.goto: net::ERR_ABORTED at https://www.trademo.com/south-korea/manufacturers/cable-wire/5 │
│ Call log: │
│ - navigating to "https://www.trademo.com/south-korea/manufacturers/cable-wire/5", waiting until "domcontentloaded" │
│ │
│ │
│ Code context: │
│ 577 response = await page.goto( │
│ 578 url, wait_until=config.wait_until, timeout=config.page_timeout │
│ 579 ) │
│ 580 redirected_url = page.url │
│ 581 except Error as e: │
│ 582 → raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}") │
│ 583 │
│ 584 await self.execute_hook( │
│ 585 "after_goto", page, context=context, url=url, response=response, config=config │
│ 586 ) │
│ 587 │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

i am still getting this error. even in
results = await crawler.arun(

[Jessi11111]

2025-04-09 13:15:45,962 - INFO - Processing URL: https://www.cabledatasheet.com/korea-south-(129)/ (Name: Korea, South (129)) [ERROR]... × https://www.trademo.com/south-korea/manufacturers/... | Error: ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ × Unexpected error in _crawl_web at line 582 in _crawl_web (.venv/lib/python3.13/site- │ │ packages/crawl4ai/async_crawler_strategy.py): │ │ Error: Failed on navigating ACS-GOTO: │ │ Page.goto: net::ERR_ABORTED at https://www.trademo.com/south-korea/manufacturers/cable-wire/5 │ │ Call log: │ │ - navigating to "https://www.trademo.com/south-korea/manufacturers/cable-wire/5", waiting until "domcontentloaded" │ │ │ │ │ │ Code context: │ │ 577 response = await page.goto( │ │ 578 url, wait_until=config.wait_until, timeout=config.page_timeout │ │ 579 ) │ │ 580 redirected_url = page.url │ │ 581 except Error as e: │ │ 582 → raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}") │ │ 583 │ │ 584 await self.execute_hook( │ │ 585 "after_goto", page, context=context, url=url, response=response, config=config │ │ 586 ) │ │ 587 │ └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

i am still getting this error. even in results = await crawler.arun(

same error today. The same code worked for me 2 days ago. when I rerun it today. It throws this error.
what could potentially be the reason?

[Jessi11111]

Unexpected error in _crawl_web at line 582 in _crawl_web (lib/python3.10/site- │
│ packages/crawl4ai/async_crawler_strategy.py): │
│ Error: Failed on navigating ACS-GOTO: │
│ Page.goto: net::ERR_INVALID_AUTH_CREDENTIALS at https://www.sample.com/ │
│ Call log: │
│ - navigating to "https://www.sample.com/", waiting until │
│ "domcontentloaded" │
│ │
│ │
│ Code context: │
│ 577 response = await page.goto( │
│ 578 url, wait_until=config.wait_until, timeout=config.page_timeout │
│ 579 ) │
│ 580 redirected_url = page.url │
│ 581 except Error as e: │
│ 582 → raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}") │
│ 583 │
│ 584 await self.execute_hook( │
│ 585 "after_goto", page, context=context, url=url, response=response, config=config │
│ 586 ) │
│ 587