Web Crawler

Functionalities:

• Crawl through a seed url
• Defined upto a given depth only
• Leverage concurrency to crawl a url

Future Enhancements

• content parser supports all media
• concurrently parse pages
• store data

Usage

go run main.go <flags>

flags:

url      seed_url to start crawling
depth    max depth to go for a url