HtmlResource downloader

HtmlResource downloader is used for efficiently download massive amount of resources (files) from websites

warning:

App strategy is very aggressive, he recursively go deep into web structure, be sure to set parameter with cognition

Al files are downloaded by a different coroutines whit the same http ktor client, it will cause a bandwidth saturation

Usage:

• Clone Repo -> Distributable
Actually you can build a distributable though gradle task!
Application needs in the same dir a parameters.json file with parameters for start the task
see DataStructure.kt/Parameters for json structure


• Clone Repo -> Run From IDE
You can set parameters in main, and if there is not a parameters.json in the mail folder, it will be created!
Remember to delete parameter.json if you want to reset them

Output:

• all files will be downloaded in "outputFolder", without lose path structure
• all skip preference (see parameteres) will be attended
• if some file throw some kind of error will be skipped and signed in skippedFile.csv, with relative error description
• detailed and annoying infos are printed on console