Some catalogue specific elements do not work properly

[TobiasNx]

Some of the changes in GKT and other hbz specific elements with letters do not validate properly even if they are configured:

qa-catalogue/src/main/java/de/gwdg/metadataqa/marc/definition/tags/hbztags/TagGKT.java

Lines 55 to 64 in 275f9cf

	"t", "Titel eines Werkes", "NR",
	"f", "Datum eines Werkes", "NR",
	"g", "Zusatz", "R",
	"h", "Inhaltstyp", "R",
	"l", "Sprache der Expression", "R",
	"m", "Medium der Musikaufführung", "R",
	"n", "Zählung eines Teils/einer Abteilung eines Werkes", "R",
	"o", "Angabe des Musikarrangements", "R",
	"p", "Titel eines Teils/einer Abteilung eines Werkes", "R",
	"r", "Tonart", "NR",

[TobiasNx]

The same happens with core specific enhancements: #13

[pkiraly]

Did you specified the local version with --marcVersion? Could you send me the parameters you applied for the validation?

[TobiasNx]

qa-catalogue/catalogues/hbz.sh

Line 14 in 275f9cf

TYPE_PARAMS="--marcVersion HBZ --marcxml --fixAlma --ignorableRecords DEL$a=Y --ignorableFields 964,940,941,942,944,945,946,947,948,949,950,951,952,955,956,957,958,959,966,967,970,971,972,973,974,975,976,977,978,978,979"

[pkiraly]

Thanks! It seems OK. The next step would be to write a unit test. If you could add a file with about 1-2 MARCXML records into src/test/resources/marc of your fork? I would write the unit tests and it helps detecting the errors.

[TobiasNx]

@pkiraly we added three marcxml files in the folder marcxml. Also I added distinct files to hint the issues in the file names.

0358af4

[pkiraly]

@TobiasNx Thanks! I wrote a unit test against these 3 files, but I was not able to reproduce the error. I found another error though: the ignorableRecords parameter throw an error when we save the parameters into a JSON file. This might block the success of the validation. Could you check the validation.log if you find a Java exception close to the end of the file?
I just fixed this issue in the main repository, see pkiraly#525.

I can not push the test against HBZ files because I do not have the necessary permission, so I put the code here. Please add it to src/test/java/de/gwdg/metadataqa/marc/cli/ValidatorCliTest.java:

// add this line to the import section
import java.util.stream.Collectors;

// put it after the last test method
  @Test
  public void validate_whenHbz() throws Exception {
    clearOutput(outputDir, outputFiles);

    ValidatorCli processor = new ValidatorCli(new String[]{
      "--schemaType", "MARC21",
      "--marcVersion", "HBZ",
      "--marcxml",
      "--outputDir", outputDir,
      "--fixAlma",
      "--ignorableRecords", "DEL$a=Y",
      "--ignorableFields", "964,940,941,942,944,945,946,947,948,949,950,951,952,955,956,957,958,959,966,967,970,971,972,973,974,975,976,977,978,978,979",
      "--details",
      "--trimId",
      "--summary",
      TestUtils.getPath("marcxml/990082522550206441_missing_validation_custom_subfield_9_core_710.xml"),
      TestUtils.getPath("marcxml/990171082050206441_missing_validation_custom_ind2_9_core_246.xml"),
      TestUtils.getPath("marcxml/991000922029706482_missing_subfield_validation_t_in_customfield_GKT.xml"),
    });

    RecordIterator iterator = new RecordIterator(processor);
    iterator.setProcessWithErrors(true);
    iterator.start();

    List<String> lines = getFileLines("issue-summary.csv");
    assertEquals(3, lines.size());
    List<String> undefinedFields = lines.stream()
      .filter(line -> line.contains("undefined field"))
      .collect(Collectors.toList());
    assertEquals(0, undefinedFields.size());
    // Pattern pattern = Pattern.compile("^\\d+,952,\\d+,\\d+,undefined field");
    // assertTrue(pattern.matcher(undefinedFields.get(0)).find());
  }

[TobiasNx]

@Phu2 will take care of it.

[Phu2]

Could you check the validation.log if you find a Java exception close to the end of the file?

No exceptions found. These are the last 20 lines of processing our whole basedump containing >27 mio records:

qa-catalogue@quaoar4:~/qa-catalogue$ tail -n 20 logs/hbz/validate.log 
Sep 23, 2024 3:27:51 PM de.gwdg.metadataqa.marc.dao.Control007 processContent
SEVERE: #991013998379706467) invalid category for 007: '|'
Sep 23, 2024 3:27:51 PM de.gwdg.metadataqa.marc.dao.Control007 processContent
SEVERE: #991013998419706467) invalid category for 007: '|'
Sep 23, 2024 3:27:51 PM de.gwdg.metadataqa.marc.dao.Control007 processContent
SEVERE: #991013998449706467) invalid category for 007: '|'
Sep 23, 2024 3:27:51 PM de.gwdg.metadataqa.marc.dao.Control007 processContent
SEVERE: #991003042719706480) invalid category for 007: '|'
Sep 23, 2024 3:27:52 PM de.gwdg.metadataqa.marc.cli.utils.RecordIterator processFile
INFO: Finished processing file. Processed 27,824,613 records.
Sep 23, 2024 3:27:52 PM de.gwdg.metadataqa.marc.cli.ValidatorCli afterIteration
INFO: printCounter
Sep 23, 2024 3:27:52 PM de.gwdg.metadataqa.marc.cli.ValidatorCli afterIteration
INFO: Saving summary
Sep 23, 2024 3:32:46 PM de.gwdg.metadataqa.marc.cli.ValidatorCli afterIteration
INFO: all printing is DONE
Sep 23, 2024 3:32:46 PM de.gwdg.metadataqa.marc.cli.QACli saveParameters
INFO: Saving configuration to /opt/qa-catalogue/output/hbz/validation.params.json.
Sep 23, 2024 3:32:46 PM de.gwdg.metadataqa.marc.cli.utils.RecordIterator start
INFO: Bye! It took: 01:40:42

[Phu2]

I can not push the test against HBZ files because I do not have the necessary permission

Now you have write access (pending invitation).

[Phu2]

@pkiraly Updated ValidatorCliTest.java as suggested, see new branch 14-test-validation.

[TobiasNx]

$ mvn clean install results in an error: https://gist.githubusercontent.com/TobiasNx/2d648828520523f7a7460dd555d57688/raw/2272d0c8c98fa6335519eb88c9d093e1adcbab90/qaError

[pkiraly]

This is what the pkiraly#525 fixes. Now I have write permission, so I will fix it today in this branch.

[pkiraly]

@TobiasNx I pushed the changes. You can try it again.

[Phu2]

Thanks, @pkiraly ! mvn test runs fine without any errors.
@TobiasNx please have a look when you are back from vacation.

[TobiasNx]

See: #17

mvn clean install locally seems to work now and the new pr seems to work too.