WIP SRUopener #682

dr0i · 2025-03-28T12:17:49Z

This is a draft and WIP.
@TobiasNx you can use it for functional testing.

Resolves #510.

TobiasNx · 2025-03-28T13:36:41Z

Nice seems to work. +1
The printed logs are a little bit esoteric:

, startRecord=1, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=1001
, startRecord=1001, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=2001
, startRecord=2001, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=3001
, startRecord=3001, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=4001
, startRecord=4001, maximumRecords=1000, istream.length=437

TobiasNx · 2025-04-10T08:55:57Z

@dr0i is still in review?

dr0i · 2025-04-10T08:59:18Z

As we found out in #510 this PR needs a complete redesign.

dr0i · 2025-06-02T14:37:44Z

@TobiasNx can you do functional tests before I go on here? Have a look at the @Description to see how it works (hint: "stream" based, i.e. other than the OAI-PMH opener works atm.)
I've added the class to flux-commands.
[edit]: and ignore the failing editorconfigChecker for now.

TobiasNx · 2025-06-04T09:54:41Z

@dr0i I tried to install the dist: https://metafacture.github.io/metafacture-documentation/docs/flux/Flux-User-Guide.html#build-from-local-distribution to try the runner for functional testing

but it runs into errors:

$ ./gradlew installDist

> Configure project :
HEAD has no annotated tags
No SCM tag found. Making a snapshot build
Feature branch found
Version is feature-510-addSruOpener-SNAPSHOT

[Incubating] Problems report is available at: file:///home/user/git/metafacture-core/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.13/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

When I test the flux.sh then it outputs the following:

$ /home/user/git/metafacture-core/metafacture-runner/build/install/metafacture-core/flux.sh
Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.metafacture.runner.Flux.main(Flux.java:62)
Caused by: org.metafacture.commons.reflection.ReflectionException: Class not found: org.metafacture.io.
        at org.metafacture.commons.reflection.ReflectionUtil.loadClass(ReflectionUtil.java:70)
        at org.metafacture.commons.reflection.ObjectFactory.loadClassesFromMap(ObjectFactory.java:57)
        at org.metafacture.flux.parser.FluxProgramm.<clinit>(FluxProgramm.java:54)
        ... 1 more
Caused by: java.lang.ClassNotFoundException: org.metafacture.io.
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
        at org.metafacture.commons.reflection.ReflectionUtil.loadClass(ReflectionUtil.java:67)
        ... 3 more

Can you help? (I tested the current master to compare, there $ ./gradlew installDist and $ /home/user/git/metafacture-core/metafacture-runner/build/install/metafacture-core/flux.sh works)

dr0i · 2025-06-05T07:43:05Z

Ah, I accidently removed the TarReader.
Try again please.

TobiasNx · 2025-06-18T07:08:48Z

Current version stucks in an endless SRU request loop starting by 1 again after finishing all request does not matter if a total number of records is given or not:

e.g.

"https://services.dnb.de/sru/authorities"
| open-sru(recordSchema="MARC21plus-xml", query="WOE%3Dsozialistenkongress%20and%20COD%3Ds",version="1.1",maximumRecords="10")
| object-batch-log(batchsize="10")
| as-records
| write(FLUX_DIR + "result.txt")
;

"https://services.dnb.de/sru/authorities"
| open-sru(recordSchema="MARC21plus-xml", query="WOE%3Dsozialistenkongress%20and%20COD%3Ds",version="1.1",maximumRecords="10",total="10")
| object-batch-log(batchsize="10")
| as-records
| write(FLUX_DIR + "result.txt")
;

Both result in, see that recordPosition 1 is turning up again after the expected last recordPosition 8:

<?xml version="1.0" encoding="UTF-8"?><searchRetrieveResponse xmlns="http://www.loc.gov/zing/srw/"><version>1.1</version><numberOfRecords>8</numberOfRecords><records><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">042278333</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20110429135047.0</controlfield>
    <controlfield tag="008">900305n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">4227833-8</subfield>
      <subfield code="0">http://d-nb.info/gnd/4227833-8</subfield>
      <subfield code="2">gnd</subfield>
    </datafield>
    <datafield ind1=" " ind2=" " tag="035">
      <subfield code="a">(DE-101)042278333</subfield>
...
    <datafield ind1=" " ind2=" " tag="913">
      <subfield code="S">swd</subfield>
      <subfield code="i">k</subfield>
      <subfield code="a">Internationaler Sozialistenkongress</subfield>
      <subfield code="0">(DE-588c)4021089-3</subfield>
    </datafield>
  </record>
</collection></recordData><recordPosition>8</recordPosition></record></records><echoedSearchRetrieveRequest><version>1.1</version><query>WOE=sozialistenkongress and COD=s</query><xQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/><startRecord>6</startRecord><maximumRecords>5</maximumRecords><recordSchema>MARC21plus-xml</recordSchema></echoedSearchRetrieveRequest></searchRetrieveResponse>
<?xml version="1.0" encoding="UTF-8"?><searchRetrieveResponse xmlns="http://www.loc.gov/zing/srw/"><version>1.1</version><numberOfRecords>8</numberOfRecords><records><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">042278333</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20110429135047.0</controlfield>
    <controlfield tag="008">900305n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">4227833-8</subfield>
      <subfield code="0">http://d-nb.info/gnd/4227833-8</subfield>
      <subfield code="2">gnd</subfield>
...
    </datafield>
    <datafield ind1=" " ind2=" " tag="913">
      <subfield code="S">swd</subfield>
      <subfield code="i">c</subfield>
      <subfield code="a">Bern / Internationaler Sozialistenkongress &lt;1919&gt;</subfield>
      <subfield code="0">(DE-588c)4227833-8</subfield>
    </datafield>
  </record>
</collection></recordData><recordPosition>1</recordPosition></record><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">1267605979</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20230329111229.0</controlfield>
    <controlfield tag="008">220908n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">1267605979</subfield>
      <subfield code="0">http://d-nb.info/gnd/1267605979</subfield>
      <subfield code="2">gnd</subfield>
...

TobiasNx

It seems that SRU opener stucks in infinite loop. See: #682 (comment)

dr0i · 2025-06-20T09:35:43Z

The inifinite loop should be fixed with b92238b, please try again @TobiasNx .

dr0i · 2025-06-27T11:44:29Z

@TobiasNx Can you update here that this is no bug but caused by marcxmlplus (or so) ?

TobiasNx · 2025-06-30T09:46:46Z

@dr0i The behaviour I reported was not the problem from the workshop. But the identation behaviour i reported is related to handle-generic-xml so it is no bug of the SRU opener.

Every single output is a valid XML by itself.

dr0i · 2025-07-10T11:31:01Z

Hi @blackwinter if you have some time: can you implement tests here? Functional-wise is the modul ready.

blackwinter

Made a first pass and left some comments. We should discuss tests after the open questions are resolved.

metafacture-io/build.gradle

blackwinter · 2025-07-22T08:58:31Z

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

Fails Checkstyle check.

Solved in 415dece besides the ClassFanOutComplexity complain. What to do about it?

Just checkstyle-disable-line it for now.

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

blackwinter · 2025-07-22T09:27:30Z

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

+            DocumentBuilder docBuilder = factory.newDocumentBuilder();
+            Document xmldoc = docBuilder.parse(inputStreamOfURl);
+
+            Transformer t = TransformerFactory.newInstance().newTransformer();


ditto (made them final)

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

metafacture-io/src/test/java/org/metafacture/io/SruOpenerTest.java

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

See #682 (comment).

Co-authored-by: Jens Wille <[email protected]>

…-core into 510-addSruOpener

- adjust test query to work with URLencoding Complements 2fd8da6. i#Bitte geben Sie eine Commit-Beschreibung für Ihre Änderungen ein. Zeilen,

blackwinter

Apart from the tests, should be mostly okay. Left some more comments regarding small optimizations and naming improvements.

P.S.: Please let the original commenter resolve any open conversations. They should have the opportunity to decide whether they're satisfied with your proposed solution.

blackwinter · 2025-10-24T11:54:20Z

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

Just checkstyle-disable-line it for now.

blackwinter · 2025-10-24T11:57:47Z

metafacture-io/src/test/java/org/metafacture/io/SruOpenerTest.java

Fails Checkstyle check.

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

blackwinter · 2025-10-24T12:09:26Z

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

+    @Override
+    public void process(final String baseUrl) {
+
+        final StringBuilder srUrl = new StringBuilder(baseUrl);


Should be a String to begin with (no need to build the string for every iteration).

Throw the exception first, then either build the string once or just concatenate all its elements.

blackwinter · 2025-10-24T12:13:53Z

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

+            try {
+                final InputStream inputStreamOfURl = retrieveUrl(srUrl);
+                final DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
+                final DocumentBuilder docBuilder = factory.newDocumentBuilder();


Can the DocumentBuilder be reused? Then move it outside the loop or make it static.

blackwinter · 2025-10-24T12:14:18Z

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

+                final DocumentBuilder docBuilder = factory.newDocumentBuilder();
+                final Document xmldoc = docBuilder.parse(inputStreamOfURl);
+
+                final Transformer t = TransformerFactory.newInstance().newTransformer();


Give t a more descriptive name.

Co-authored-by: Jens Wille <[email protected]>

github-project-automation bot added this to Metafacture Mar 28, 2025

dr0i requested a review from TobiasNx March 28, 2025 12:18

dr0i changed the title ~~WIP SRUopener (#510)~~ WIP SRUopener Mar 28, 2025

dr0i moved this to Review in Metafacture Mar 28, 2025

dr0i added Enhancement Flux Conform To Catmandu labels Mar 28, 2025

TobiasNx approved these changes Mar 28, 2025

View reviewed changes

TobiasNx mentioned this pull request Mar 31, 2025

Add SRU opener / open-sru #510

Open

dr0i force-pushed the 510-addSruOpener branch from ecd9c8c to c3f3ad6 Compare April 10, 2025 13:32

dr0i force-pushed the 510-addSruOpener branch from 84d6845 to 3dc0416 Compare June 2, 2025 14:24

dr0i requested a review from TobiasNx June 2, 2025 14:38

dr0i assigned TobiasNx Jun 2, 2025

TobiasNx assigned dr0i and unassigned TobiasNx Jun 4, 2025

dr0i assigned TobiasNx and unassigned dr0i Jun 5, 2025

dr0i force-pushed the 510-addSruOpener branch from 4a00838 to ac80718 Compare June 10, 2025 13:02

TobiasNx assigned dr0i and unassigned TobiasNx Jun 18, 2025

TobiasNx requested changes Jun 20, 2025

View reviewed changes

dr0i assigned TobiasNx Jun 20, 2025

dr0i added 2 commits July 10, 2025 13:27

Add SruOpener (#510)

5aa1762

Every single output is a valid XML by itself.

WIP add test (#510)

65e7592

dr0i force-pushed the 510-addSruOpener branch from 841fae3 to 65e7592 Compare July 10, 2025 11:28

dr0i assigned blackwinter and unassigned dr0i Jul 10, 2025

dr0i moved this from Working to Selected in Metafacture Jul 10, 2025

blackwinter requested changes Jul 22, 2025

View reviewed changes

blackwinter assigned dr0i and unassigned blackwinter Jul 22, 2025

dr0i and others added 12 commits September 11, 2025 16:02

Fix loop when getting data

ecd05fe

Remove unused dependecies

406c5ce

See #682 (comment).

Adjust to fulfill checkstyle conditions

415dece

Fix license

a111aff

Use int in setters

afc756c

Update metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

94da562

Co-authored-by: Jens Wille <[email protected]>

URLencode query parameter

2fd8da6

Merge branch '510-addSruOpener' of github.com:metafacture/metafacture…

6e7b03c

…-core into 510-addSruOpener

Use try to invoke auto closeable

7f7beab

Inline method thus getting rid of two member variables

1291754

Fix test endless loop

907fdfc

- adjust test query to work with URLencoding Complements 2fd8da6. i#Bitte geben Sie eine Commit-Beschreibung für Ihre Änderungen ein. Zeilen,

Set maximumRecords earlier as it is invariant

aa161ce

dr0i assigned blackwinter and unassigned dr0i Oct 20, 2025

blackwinter requested changes Oct 24, 2025

View reviewed changes

blackwinter assigned dr0i and unassigned blackwinter Oct 24, 2025

Update metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

ad7ae5f

Co-authored-by: Jens Wille <[email protected]>

WIP SRUopener #682

Are you sure you want to change the base?

WIP SRUopener #682

Conversation

dr0i commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiasNx commented Mar 28, 2025

Uh oh!

TobiasNx commented Apr 10, 2025

Uh oh!

dr0i commented Apr 10, 2025

Uh oh!

dr0i commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiasNx commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr0i commented Jun 5, 2025

Uh oh!

TobiasNx commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiasNx left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dr0i commented Jun 20, 2025

Uh oh!

dr0i commented Jun 27, 2025

Uh oh!

TobiasNx commented Jun 30, 2025

Uh oh!

dr0i commented Jul 10, 2025

Uh oh!

blackwinter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blackwinter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dr0i commented Mar 28, 2025 •

edited

Loading

dr0i commented Jun 2, 2025 •

edited

Loading

TobiasNx commented Jun 4, 2025 •

edited

Loading

TobiasNx commented Jun 18, 2025 •

edited

Loading

TobiasNx left a comment •

edited

Loading