Skip to content

NUTCH-3110 Upgrade to Tika 3.2.3 - #887

Merged
sebastian-nagel merged 3 commits into
apache:masterfrom
lewismc:NUTCH-3110
Feb 3, 2026
Merged

sebastian-nagel merged 3 commits into
apache:masterfrom
lewismc:NUTCH-3110

Conversation

@lewismc

@lewismc lewismc commented Jan 12, 2026 •

Copy link
Copy Markdown
Member

This PR is an attempt to address NUTCH-3110 and in the process supersede #850.
Essentially it upgrades Apache Tika from the shaded artifacts to the official Tika 3.2.3 release, addressing compatibility issues and restoring full functionality. Some noteworthy proposals

  • Both plugins (language-identifier & parse-tika) exclude slf4j-api to prevent class loader conflicts (NUTCH-3108)
  • Duplicate outlinks: Changed HashMap to LinkedHashMap in DOMContentUtils.java to preserve link insertion order while deduplicating.
  • UTF-16 encoding test: Fixed double BOM issue in TestHtmlParser.java where Java's UTF-16 encoder was adding a second BOM.
    Boilerpipe support: Restored boilerpipe content extraction using the new tika-handler-boilerpipe module.

Additionally a bunch of new tests will assist in future Tika upgrades

  • TestBoilerpipeExtraction - Boilerpipe integration tests
  • TestLinkExtractionEdgeCases - Link extraction behavior tests
  • TestEncodingDetection - Charset detection tests
  • TestMetadataExtraction - HTML metadata extraction tests
  • TestParserFailureHandling - Error handling/graceful degradation tests

This PR needs diligent testing in distributed mode before we consider merging.

sebastian-nagel and others added 3 commits March 28, 2025 09:19
Upgrade to shaded Tika packages 3.1.0.0 provided by Tim Allison.
The shaded packages are required to avoid version conflicts when
running in distributed mode caused by incompatible versions of
the commons-io jar shipped with Hadoop and required by Tika,
cf. NUTCH-2959.
Add "text/javascript" as MIME type supported by "parse-js".
Note: fixes parse-js unit tests. Tika 3.1.0 identifies
the Javascript test document as "text/javascript" instead of
"application/javascript".

@sebastian-nagel sebastian-nagel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @lewismc! Great work! Especially, the new unit tests.

I'm going to merge the PR.

Successfully tested the PR in pseudo-distributed mode on Hadoop 3.4.2

  • testing Nutch tools: fetch, updatedb, etc.
  • fetching and parsing Tika test documents using this script, results in comparison to the previously used Tika 2.9.1.0, see below.
success failed total
649 92 741 tika-2.9.1.0-shaded
654 85 741 tika-3.2.3

Results have improved. The errors are because of encrypted documents or MIME types not supported by the "tika-parsers-standard-package":

     15  Unable to process: document is encrypted
      7  Can't retrieve Tika parser for mime-type application/rdf+xml
      3  Can't retrieve Tika parser for mime-type application/x-x509-key; format=der
      3  Can't retrieve Tika parser for mime-type application/x-x509-key; format=pem
      3  Can't retrieve Tika parser for mime-type text/x-matlab
      2  Can't retrieve Tika parser for mime-type application/x-matroska
      2  Can't retrieve Tika parser for mime-type application/x-quattro-pro
      2  Can't retrieve Tika parser for mime-type application/x-rar-compressed; version=4
      2  Can't retrieve Tika parser for mime-type application/x-sas
      2  Can't retrieve Tika parser for mime-type text/javascript
      2  Can't retrieve Tika parser for mime-type text/x-csrc
      1  Can't retrieve Tika parser for mime-type application/octet-stream
      1  Can't retrieve Tika parser for mime-type application/vnd.google-earth.kmz
      1  Can't retrieve Tika parser for mime-type application/vnd.ms-cab-compressed
      1  Can't retrieve Tika parser for mime-type application/vnd.openofficeorg.autotext
      1  Can't retrieve Tika parser for mime-type application/vnd.openofficeorg.extension
      1  Can't retrieve Tika parser for mime-type application/vnd.syncml.dmddf+xml
      1  Can't retrieve Tika parser for mime-type application/vnd.wordperfect; version=4.2
      1  Can't retrieve Tika parser for mime-type application/x-apple-diskimage
      1  Can't retrieve Tika parser for mime-type application/x-gtar
      1  Can't retrieve Tika parser for mime-type application/x-hwp
      1  Can't retrieve Tika parser for mime-type application/x-msdownload; format=pe32
      1  Can't retrieve Tika parser for mime-type application/x-quattro-pro; version=1+5
      1  Can't retrieve Tika parser for mime-type application/x-quattro-pro; version=1-4
      1  Can't retrieve Tika parser for mime-type application/x-quattro-pro; version=5
      1  Can't retrieve Tika parser for mime-type application/x-quattro-pro; version=6
      1  Can't retrieve Tika parser for mime-type application/x-sas-data-v6
      1  Can't retrieve Tika parser for mime-type application/x-sas-xport
      1  Can't retrieve Tika parser for mime-type application/x-snappy-framed
      1  Can't retrieve Tika parser for mime-type application/x-webarchive
      1  Can't retrieve Tika parser for mime-type application/x-x509-cert; format=der
      1  Can't retrieve Tika parser for mime-type application/x-x509-cert; format=pem
      1  Can't retrieve Tika parser for mime-type application/x-x509-dsa-parameters
      1  Can't retrieve Tika parser for mime-type application/x-x509-ec-parameters
      1  Can't retrieve Tika parser for mime-type application/zstd
      1  Can't retrieve Tika parser for mime-type image/avif
      1  Can't retrieve Tika parser for mime-type image/jp2
      1  Can't retrieve Tika parser for mime-type image/vnd.dgn; version=7
      1  Can't retrieve Tika parser for mime-type image/x-portable-pixmap
      1  Can't retrieve Tika parser for mime-type message/x-emlx
      1  Can't retrieve Tika parser for mime-type multipart/related
      1  Can't retrieve Tika parser for mime-type text/x-vcalendar
      1  Can't retrieve Tika parser for mime-type video/x-matroska
      1  Could not create CompressorInputStream.
      1  Couldn't find core part
      1  DRM encrypted document is not yet supported by Apache POI
      1  Expecting space or asterisk at beginning of record, not:10
      1  TIKA-198: Illegal IOException from org.apache.tika.parser.pkg.PackageParser@45b071e0
      1  Unable to unpack document stream
      1  Unexpected RuntimeException from org.apache.tika.parser.microsoft.chm.ChmParser@160c9278
      1  Unexpected RuntimeException from org.apache.tika.parser.microsoft.ooxml.OOXMLParser@2b5413ad
      1  UnsupportedZipFeature
      1  cannot parse detached pkcs7 signature (no signed data to parse)

@sebastian-nagel
sebastian-nagel merged commit 3101a9e into apache:master Feb 3, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants