Collate bug: loss of tokens during collation

[matgille]

Hi,

I'm facing a loss of tokens with the following json data (juxtaposition_2.json), after collation.
juxtaposition_6.json

Token rmtBBTT is lost during process, and I don't know exactly why. The number of witnesses seems to have some influence on the result.
juxtaposition_6_coll.json

The data is of poor quality (it's HTR output and I want to use collation for correction purposes) but no token should disappear during the collation process...

The test code used to collate this data is the following:

import collatex
import json

with open("juxtaposition_6.json") as input_json:
  collatex_dict = json.load(input_json)
collation_table = collatex.collate(collation=collatex_dict, output="json", segmentation=False, near_match=True)


with open("juxtaposition_6_coll.json", "w") as output_json:
  output_json.write(collation_table)

I'll try to investigate when I have some time. Any clue on where to look ?

Best,

Matthias

[matgille]

Edit: with near_match set to False, the problem disapears.

[djbpitt]

Thank you, @matgille , for reporting this. Could you please create a small reproducible example? The current input file has a very large number of tokens, which makes it challenging to trace the flow.

[djbpitt]

@matgille @rhdekker Confirming the same behavior in https://github.com/djbpitt/mikulka (see tag https://github.com/djbpitt/mikulka/releases/tag/CollateX-near-match-token-loss-bug).

With near_match set to True some tokens are lost; with near_match set to False they are retained. In particular, "съ" is lost before "единеинемъ" in witness A and adjacent witnesses with:

collated = collate(json.loads(jsonInput), segmentation=False, near_match=True, output="html" )

The tokens apparently are not lost if near_match is set to False.

This appears to be a bug in the near-match routine.