What is the usecase of this?

[samyogdhital]

Hello @thadius83 what is the usecase of this?
I know what firecrawl does. I let's you scrape and crawl website.

So what does this firecrawl-customgoogle do?

It it useful for doing serp query through google?
And why do we need soo many (8-10) google_api_key?

I did google search and there is google programmable search. That let's you do search query through api but it is only to 10k queries a day.

So what is the exact usecase this repo is solving that firecrawl does not do?
Is it only modified for doing google serp query when using their /search endpoint?

[samyogdhital]

What I want to do is I want to use Google to search different queries, get the urls and then scrape theses url that you get through Firecrawl. Does this repo do this precisely?

[thadius83]

Hello @thadius83 what is the usecase of this? I know what firecrawl does. I let's you scrape and crawl website.

So what does this firecrawl-customgoogle do?

It it useful for doing serp query through google? And why do we need soo many (8-10) google_api_key?

I did google search and there is google programmable search. That let's you do search query through api but it is only to 10k queries a day.

So what is the exact usecase this repo is solving that firecrawl does not do? Is it only modified for doing google serp query when using their /search endpoint?

Because the default setup on the self hosted version of firecrawl is blocked by google, you’ll end up getting rate limited with 429 errors.

My fork adds support for google API & CSE/programmable search engine access for SERP queries.

This function can also extract the pages returned from the SERP query.

I had a use case for up to 10 keys. But it works with just one.

[thadius83]

What I want to do is I want to use Google to search different queries, get the urls and then scrape theses url that you get through Firecrawl. Does this repo do this precisely?

Yes it does.

I’ll get some example output uploaded shortly, and some screenshots of the google API charts.

[samyogdhital]

@thadius83 Thanks for the quick reply.

My fork adds support for google API & CSE/programmable search engine access for SERP queries.
Got it thanks and I am following the repo. Would love to have quick examples and ss.

Thanks.
Didn't knew I would need this badly.