feat: add docs crawler

[th3w1zard1]

Add a docs crawler using to implement the following crawlers:

• ChromiumCrawler
• ChromiumCrawler
• DefaultCrawler
• specific implementation for github.

following major packages used:

• vectordb
• Puppetteer

This PR is highly inspired by Continue.dev's implementation, and the following resources/topics:

• https://stackoverflow.com/questions/50218402/nodejs-web-crawling-with-node-crawler-or-simplecrawler
• https://www.reddit.com/r/LangChain/comments/1g8scol/efficient_web_crawling_for_keeping_vector/
• https://stackoverflow.com/questions/50154133/how-to-crawl-all-the-internal-urls-of-a-website-using-crawler

[th3w1zard1]

currently only the vscode side is implemented. A brainstorming session is needed to determine best way to integrate with aider

[lee88688]

this looks good idea. but I think this may not be the good as continue.dev. because this extension use aider as backend, it will have no control of what the system or other message is. so the plugin may not have enough way to optimize the result.
do you have any ideas about this?

[th3w1zard1]

I don't think the system message would be relevant: what would be important is relevant documentation being pulled and appended to the user prompt?

The system message I can't imagine why it would be relevant.

The main issue i see is the documentation retrieval should be created through aider and then an interface provided to this extension

[lee88688]

Aider's core code is not provide a general interface for extension, some features may be a little hard or tricky to implement.

This may be a good idea,but may not a good for now. If the extension want to go forward, we need aider 's support as well.

Otherwise this extension needs to implement all backend ourselves.