Skip to content

Reintegrate all skyshard changes #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
rborer wants to merge 152 commits into
base: smallrivers
Choose a base branch
from
Open

Reintegrate all skyshard changes #10

rborer wants to merge 152 commits into from

Conversation

@rborer
Copy link

@rborer rborer commented Aug 12, 2016

Cherry-pick everything until commit 0eedcaa.

andresp99999 and others added 30 commits August 9, 2016 17:24
@andresp99999
Add additional heuristics to improve title extraction
769ad82 
# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
#	src/test/java/de/jetwick/snacktory/ArticleTextExtractorTest.java
@andresp99999
Add extraction test case
57660e1
@andresp99999
Several improvements to date extraction, add additional supported dat…
80ad961 
…e patterns and test cases

# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
#	src/test/java/de/jetwick/snacktory/ArticleTextExtractorTest.java
@andresp99999
Add new test cases
a833138 
# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
@andresp99999
Include great-grand-children nodes in the weight calculation, refacto…
311d367 
…r logging info for debugging

# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
@andresp99999
Add new test case
a7e9e3e
@andresp99999
Rename test file
6791a27
@andresp99999
Give more weight to grand and greatgrand children, add additional tes…
0e753c2 
…t cases
@andresp99999
Remove comment
9e56da0
Adapt title extraction of new tests
f0aeb55 
New tests introduced by 0e753c2 requires changes as we extract the title differently than upstream.
@andresp99999
Add additional test cases for cnn.com
9d02aca 
# Conflicts:
#	src/test/java/de/jetwick/snacktory/ArticleTextExtractorTest.java
@andresp99999
Add support to disable SSL verification
2f4420c 
# Conflicts:
#	src/main/java/de/jetwick/snacktory/HtmlFetcher.java
@andresp99999
Add new test case
66468e3 
# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
Fix title extraction with HTML entities
f9e8291
@andresp99999
Add checks to support when dates are malformed
2c4dc44
@andresp99999
Add another case for date extraction
3ec9245
@andresp99999
Add support to detect partial dates in the URL
d1a9c35 
# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
@andresp99999
Add more date patterns
4f94e2a
@andresp99999
Yet more date patterns
9b175ae
@andresp99999
Fix expected value in test
895c07f
@andresp99999
Add additional use cases for date extraction
2ea6197
@andresp99999
Add additional use cases for date extraction
15c0b77
@andresp99999
Add additional use cases for date extraction
7167739
@andresp99999
Some fixes for date extraction
d8e0dc0
@andresp99999
Fix issue with some rules too generic and were matching random dates …
00edaee 
…in the articles
@andresp99999
Change rule to use less generic selectors
da6b726
@andresp99999
More date extraction fixes
87c4bbd
Another tests fix as we extract title differently
7e4bf90
@andresp99999
Support month by names in URL date extractor
5dfd227
@andresp99999
Add another date pattern
d2e2e75
andresp99999 and others added 30 commits August 12, 2016 11:12
@andresp99999
Add another extraction case
4a7b4de 
# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
@andresp99999
Add another extraction case
d8ac97e
@andresp99999
Add another extraction case
6cbc1cd 
# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
@andresp99999
Add new date extraction case
5665ef8
@andresp99999
Add a couple date extraction cases
ce8ca75
@andresp99999
yet another extraction case
83606ee 
# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
Fix tests for title extraction
9f6fef3
@andresp99999
yet another extraction case
c93253c
@andresp99999
yet another extraction case
8330a33
@andresp99999
Add another extraction case
b55de36 
# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
Fix unit test for title extraction
69a80aa
@andresp99999
Add extraction case
5436fc7 
# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
@andresp99999
Add custom extraction case for theverge
114f0ed
@andresp99999
Add custom rules for iheart.com
52160d5 
# Conflicts:
#	src/main/java/de/jetwick/snacktory/ArticleTextExtractor.java
Another title extraction test fix
852f284
@andresp99999
If the author name is smaller than 8 characters don't try to search f…
4b6bf66 
…or it to get the author_desc
Fix testBizJournal test
c2bc38a 
We get slightly less content with our implementation (probably because of karussell#47).
Fix testLeFigaroSport unit test
7580c7a
Fix testCNBC2 unit test (get the latest HTML from the website)
d168751
Bump dependencies
bba5ca2
Fixes for HtmlFetcherIntegrationTest
b742424
Fix testBBCNoCSS (get most recent content from website)
efbf1cf
Fix testData6 extracted content
7ac58c8
Fix testKdwb
7c0ce3f 
Photo credits are now excluded
Fix remaining failing unit tests
42d74d7
Better usage of assertion framework inside tests
7990909
Quick code cleanup
d327692 
Remove unused import & variables. Make sure internal classes are static so they don't keep a reference to the parent instance!
Bump maven configuration
09e8b5f
Huge changes, good opportunity to work towards a 1.4.0 release
70bae52
Merge branch 'smallrivers' into skyshard-merge
7f97aac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@rborer @andresp99999