Write a java crawler that crawls the web for data. The crawler pulls data and stores it locally. The crawler will not do any analysis on the data.
Use maven and java to write the crawler. Your crawler should be executed as followed:
java -jar crawler.jar -d <depth> -u <url>
Depth is how many level (links) deep your crawler will execute a crawl before returning. Url can be any url from which it will crawl.
The craw should save data to a local storage. The implementation of the storage layer is open to your option.
Your team will be graded as followed:
- Crawling correctness
- Storage of data to local site