Skip to content

Commit 30616ea

Browse files
committed
bump up
1 parent 9dfc185 commit 30616ea

3 files changed

Lines changed: 63 additions & 6 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Changelog
2+
3+
## 0.3.0 (unreleased)
4+
5+
The first release since 2021, the internals were rewritten along the way.
6+
7+
### Breaking
8+
9+
* Middleware is a class with a `call(hash, fields)` method now and is declared by
10+
its name, `middleware "Debug", "Save"`. The `Middleware.build` chain of the
11+
previous release is gone.
12+
* `timeout` and `ferrum` settings are replaced by `driver`, e.g.
13+
`driver :ferrum, timeout: 30`.
14+
* `intercept` is replaced by `blacklist` and `whitelist`.
15+
* Urls are visited only once by default, pass `once: false` to a request to
16+
visit one again.
17+
18+
### Added
19+
20+
* Pluggable drivers, `:ferrum` (default, real Chrome) and `:mechanize` (plain
21+
HTTP, no browser), plus a `Vessel::Driver` base class for your own.
22+
* Fields api, `field :name, value: ...` with blocks, `service: true`, `rename:`
23+
and `Vessel::Cargo::FieldType` for per name normalization.
24+
* Requests carry `data`, and accept per request `handler`, `headers`, `cookies`,
25+
`delay` and `once`.
26+
* `start_urls` accepts a hash of `url => handler`.
27+
* Cookies, `cookie`/`cookies` to set them and `allow_cookies` to control whether
28+
the ones a page sends back are kept for the next requests.
29+
* Proxy rotation, `Vessel::RoundRobinProxy` and `Vessel::ShuffledProxy`.
30+
* Retries of the requests that fail with a network error,
31+
`network_error_attempts`.
32+
* Callbacks, `before_start`, `before`, `after_change`, `info`, `after`,
33+
`before_stop` and `on_error`.
34+
* `Vessel.page_snapshot` to run the selectors against a Nokogiri document
35+
instead of the live page.
36+
* CLI and a project skeleton, `vessel new`, `generate`, `list`, `settings`,
37+
`start`, `parse` and `version`.
38+
* `Vessel::Cargo` is the crawler base class, `Vessel::Crawler` is kept as an
39+
alias.
40+
41+
### Fixed
42+
43+
* The stats timer task is shut down when a run finishes, it used to keep logging
44+
and hold a thread for the life of the process ([#36](https://github.com/rubycdp/vessel/issues/36)).
45+
* An url is registered before its page is opened, threads racing for the same
46+
url no longer visit it in parallel.
47+
* The skeleton of a new project is packaged completely, `vessel new` used to
48+
generate a project without `crawlers`, `log`, `config/fields` and
49+
`lib/helpers` when Vessel was installed as a gem
50+
([#37](https://github.com/rubycdp/vessel/issues/37)).
51+
* Settings are deep cloned into the subclasses of a crawler.
52+
* The engine is idle only when every scheduled request and item is done.
53+
* Ferrum 0.15 and up with a proxy.
54+
* Minimum Ruby is 2.7.
55+
56+
## 0.2.0 (2021-03-09)
57+
58+
* Initial public releases, see the git history for the details.

‎lib/vessel/version.rb‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# frozen_string_literal: true
22

33
module Vessel
4-
VERSION = "0.2.0"
4+
VERSION = "0.3.0"
55
end

‎vessel.gemspec‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,18 +8,16 @@ Gem::Specification.new do |s|
88
s.platform = Gem::Platform::RUBY
99
s.authors = ["Dmitry Vorotilin"]
1010
s.email = ["d.vorotilin@gmail.com"]
11-
s.homepage = "https://github.com/route/vessel"
11+
s.homepage = "https://github.com/rubycdp/vessel"
1212
s.summary = "High-level web crawling framework"
1313
s.description = "Vessel is a high-level web crawling framework, used to crawl websites and "\
1414
"extract structured data from their pages"
1515
s.license = "MIT"
16-
# Both ways of listing the files have to keep the dot files of the skeleton,
17-
# they are the ones holding its otherwise empty directories.
1816
s.files = Dir.chdir(__dir__) do
19-
tracked = `git ls-files -z bin lib LICENSE README.md`.split("\x0")
17+
tracked = `git ls-files -z bin lib CHANGELOG.md LICENSE README.md`.split("\x0")
2018
next tracked unless tracked.empty? # built outside of a git checkout
2119

22-
Dir.glob("{bin/*,lib/**/*,LICENSE,README.md}", File::FNM_DOTMATCH)
20+
Dir.glob("{bin/*,lib/**/*,CHANGELOG.md,LICENSE,README.md}", File::FNM_DOTMATCH)
2321
.grep_v(%r{/\.\.?\z})
2422
end
2523
s.bindir = "bin"
@@ -28,6 +26,7 @@ Gem::Specification.new do |s|
2826
s.metadata = {
2927
"homepage_uri" => "https://vessel.rubycdp.com/",
3028
"bug_tracker_uri" => "https://github.com/rubycdp/vessel/issues",
29+
"changelog_uri" => "https://github.com/rubycdp/vessel/blob/main/CHANGELOG.md",
3130
"documentation_uri" => "https://github.com/rubycdp/vessel/blob/main/README.md",
3231
"source_code_uri" => "https://github.com/rubycdp/vessel",
3332
"rubygems_mfa_required" => "true"

0 commit comments

Comments
 (0)