|
| 1 | +# Changelog |
| 2 | + |
| 3 | +## 0.3.0 (unreleased) |
| 4 | + |
| 5 | +The first release since 2021, the internals were rewritten along the way. |
| 6 | + |
| 7 | +### Breaking |
| 8 | + |
| 9 | +* Middleware is a class with a `call(hash, fields)` method now and is declared by |
| 10 | + its name, `middleware "Debug", "Save"`. The `Middleware.build` chain of the |
| 11 | + previous release is gone. |
| 12 | +* `timeout` and `ferrum` settings are replaced by `driver`, e.g. |
| 13 | + `driver :ferrum, timeout: 30`. |
| 14 | +* `intercept` is replaced by `blacklist` and `whitelist`. |
| 15 | +* Urls are visited only once by default, pass `once: false` to a request to |
| 16 | + visit one again. |
| 17 | + |
| 18 | +### Added |
| 19 | + |
| 20 | +* Pluggable drivers, `:ferrum` (default, real Chrome) and `:mechanize` (plain |
| 21 | + HTTP, no browser), plus a `Vessel::Driver` base class for your own. |
| 22 | +* Fields api, `field :name, value: ...` with blocks, `service: true`, `rename:` |
| 23 | + and `Vessel::Cargo::FieldType` for per name normalization. |
| 24 | +* Requests carry `data`, and accept per request `handler`, `headers`, `cookies`, |
| 25 | + `delay` and `once`. |
| 26 | +* `start_urls` accepts a hash of `url => handler`. |
| 27 | +* Cookies, `cookie`/`cookies` to set them and `allow_cookies` to control whether |
| 28 | + the ones a page sends back are kept for the next requests. |
| 29 | +* Proxy rotation, `Vessel::RoundRobinProxy` and `Vessel::ShuffledProxy`. |
| 30 | +* Retries of the requests that fail with a network error, |
| 31 | + `network_error_attempts`. |
| 32 | +* Callbacks, `before_start`, `before`, `after_change`, `info`, `after`, |
| 33 | + `before_stop` and `on_error`. |
| 34 | +* `Vessel.page_snapshot` to run the selectors against a Nokogiri document |
| 35 | + instead of the live page. |
| 36 | +* CLI and a project skeleton, `vessel new`, `generate`, `list`, `settings`, |
| 37 | + `start`, `parse` and `version`. |
| 38 | +* `Vessel::Cargo` is the crawler base class, `Vessel::Crawler` is kept as an |
| 39 | + alias. |
| 40 | + |
| 41 | +### Fixed |
| 42 | + |
| 43 | +* The stats timer task is shut down when a run finishes, it used to keep logging |
| 44 | + and hold a thread for the life of the process ([#36](https://github.com/rubycdp/vessel/issues/36)). |
| 45 | +* An url is registered before its page is opened, threads racing for the same |
| 46 | + url no longer visit it in parallel. |
| 47 | +* The skeleton of a new project is packaged completely, `vessel new` used to |
| 48 | + generate a project without `crawlers`, `log`, `config/fields` and |
| 49 | + `lib/helpers` when Vessel was installed as a gem |
| 50 | + ([#37](https://github.com/rubycdp/vessel/issues/37)). |
| 51 | +* Settings are deep cloned into the subclasses of a crawler. |
| 52 | +* The engine is idle only when every scheduled request and item is done. |
| 53 | +* Ferrum 0.15 and up with a proxy. |
| 54 | +* Minimum Ruby is 2.7. |
| 55 | + |
| 56 | +## 0.2.0 (2021-03-09) |
| 57 | + |
| 58 | +* Initial public releases, see the git history for the details. |
0 commit comments