Vertical layout not working with json output

[tarrinw]

alignment_table = collate(json.loads(json_input), output="json", layout="vertical", segmentation=False, near_match=True)

produces the exact same output as

alignment_table = collate(json.loads(json_input), output="json", layout="horizontal", segmentation=False, near_match=True)

or

alignment_table = collate(json.loads(json_input), output="json", segmentation=False, near_match=True)

But should produce output equivalent to the old java interface.

[djbpitt]

@tarrinw Can you please share a small JSON input along with:

1. The command you give to the Java version and the JSON output it returns (corresponding to the output that you want).
2. The command you give to the Python version and the JSON output that it returns (which, if I've understood correctly, is not what you expect).

As far as I can see from the documentation, the Java version supports JSON output, but does not allow control over whether the JSON output table is horizontal or vertical. Control over table orientation(horizontal vs vertical) is available for table output (plain-text table and HTML table) in the Python version, but:

• The documentation of the Java API at https://collatex.net/doc/ §7 doesn't appear to say anything about supporting user control over table orientation.
• The documentation of output options for the Python version at https://nbviewer.org/github/DiXiT-eu/collatex-tutorial/blob/master/unit5/3_collation-outputs.ipynb doesn't appear to say anything about supporting user control over table orientation for JSON output.

Here’s a bit of history: The non-JSON visualizations were intended to be suitable for final-form output options, while the JSON output was not considered a visualization; it was witness-by-witness token-based alignment table in JSON format that was intended for post-processing once it was beyond the control of CollateX. Because the JSON output was regarded as the output of the collation process, and not as a visualization of the result, the specific details of the tabular shape of the JSON output are part of the Gothenburg model (described in the CollateX documentation at https://collatex.net/doc/#json-output). This allows an application to make an API call to any Gothenburg-conformant collation tool (such as CollateX) and know the shape of the JSON to be returned. This has at least the following two implications:

1. Users were not intended to have control over the shape of the JSON output as it emerges from CollateX (e.g., whether the table is horizontal or vertical), although users will do whatever they want and need to do with it once CollateX emits it.
2. The Java and Python versions of CollateX should produce the same JSON output. If that isn't the case, it's a divergence from the Gothenburg model and a bug in the implementation.

Unfortunately, that JSON output is not a visualization is not immediately clear from the available documentation. There was no final Interedition White Paper that documented the Gothenburg model, and although the CollateX documentation (see above) is explicit about what JSON output should look like, it isn't clear whether that is a feature of CollateX or an output format mandated by the Gothenburg model. It's the latter, but in the absence of a White Paper there's no way for a user to know that. Knowing what we know now, we would have documented this detail, and, for what it's worth, we might have selected a different JSON format (such as one based on the Rhine delta / variant graph, rather than the alignment table), but those details were not as clear in 2009 as they are in 2025.

[tarrinw]

This isn't a big problem - as you write the json is for post-processing and I have just readjusted it in my code to the vertical structure - but the documentation (https://interedition.github.io/collatex/pythonport.html) explicitly states that you can produce vertical layout for the json output:

"By default the output produces is equivalent to the horizontal alignment tables above. If the layout option is set to vertical then the export will be equivalent to the vertical alignment tables above. The latter produces a format which is structurally the same as the json output option from the Java microservices version of collateX. The only difference between the two is that in the Python export None is used for empty cells and in the Java microservices an empty array is used."

And it would make sense to be able to produce the same output as in java.

Sample code:

json_input = '''
{
  "witnesses": [
    {
      "id": "manual",
      "tokens": [
        {
          "id": "3676991",
          "t": "verþ"
        },
        {
          "id": "3676992",
          "t": "lꜹnıþ"
        },
        {
          "id": "3676993",
          "t": "aꞇ"
        },
        {
          "id": "3676994",
          "t": "vılıa"
        },
        {
          "id": "3676995",
          "t": "oso̅"
        },
        {
          "id": "3676997",
          "t": "glępr"
        }
      ]
    },
    {
      "id": "htr",
      "tokens": [
        {
          "id": "12340286",
          "t": "ꝼ"
        },
        {
          "id": "12340287",
          "t": "G"
        },
        {
          "id": "12340288",
          "t": "verþ"
        },
        {
          "id": "12340289",
          "t": "laꝼnıþ"
        },
        {
          "id": "12340290",
          "t": "aꞇ"
        },
        {
          "id": "12340291",
          "t": "vılıa"
        },
        {
          "id": "12340292",
          "t": "ogo̅"
        },
        {
          "id": "12340293",
          "t": "gle"
        }
      ]
    }
  ]
}
'''

import json
from collatex import *
collation = Collation()

alignment_table = collate(json.loads(json_input), output="json", segmentation=False, near_match=True, layout="vertical")
print(alignment_table)

Gives only horizontal output (with or without layout="vertical"):

{"table": [[null, null, [{"_sigil": "manual", "_token_array_position": 0, "id": "3676991", "t": "verþ"}], [{"_sigil": "manual", "_token_array_position": 1, "id": "3676992", "t": "lꜹnıþ"}], [{"_sigil": "manual", "_token_array_position": 2, "id": "3676993", "t": "aꞇ"}], [{"_sigil": "manual", "_token_array_position": 3, "id": "3676994", "t": "vılıa"}], [{"_sigil": "manual", "_token_array_position": 4, "id": "3676995", "t": "oso̅"}], [{"_sigil": "manual", "_token_array_position": 5, "id": "3676997", "t": "glępr"}]], [[{"_sigil": "htr", "_token_array_position": 7, "id": "12340286", "t": "ꝼ"}], [{"_sigil": "htr", "_token_array_position": 8, "id": "12340287", "t": "G"}], [{"_sigil": "htr", "_token_array_position": 9, "id": "12340288", "t": "verþ"}], [{"_sigil": "htr", "_token_array_position": 10, "id": "12340289", "t": "laꝼnıþ"}], [{"_sigil": "htr", "_token_array_position": 11, "id": "12340290", "t": "aꞇ"}], [{"_sigil": "htr", "_token_array_position": 12, "id": "12340291", "t": "vılıa"}], [{"_sigil": "htr", "_token_array_position": 13, "id": "12340292", "t": "ogo̅"}], [{"_sigil": "htr", "_token_array_position": 14, "id": "12340293", "t": "gle"}]]], "witnesses": ["manual", "htr"]}

Via java as microservice (the input json in tmp.json file):

curl --data '@tmp.json' http://localhost:7369/collate

gives vertical by default:

{"witnesses":["htr","manual"],"table":[[[{"id":"12340286","t":"ꝼ"},{"id":"12340287","t":"G"}],[]],[[{"id":"12340288","t":"verþ"}],[{"id":"3676991","t":"verþ"}]],[[{"id":"12340289","t":"laꝼnıþ"}],[{"id":"3676992","t":"lꜹnıþ"}]],[[{"id":"12340290","t":"aꞇ"},{"id":"12340291","t":"vılıa"}],[{"id":"3676993","t":"aꞇ"},{"id":"3676994","t":"vılıa"}]],[[{"id":"12340292","t":"ogo̅"},{"id":"12340293","t":"gle"}],[{"id":"3676995","t":"oso̅"},{"id":"3676997","t":"glępr"}]]]}

[djbpitt]

@tarrinw Thank you for the follow-up. We can confirm that:

1. The Java and Python JSON outputs differ. In Python the table property is an array of witnesses (one array member per witness), while in Java the table property is an array of what we are now calling "alignment points", each of which contains an array of witness data for that alignment point.
2. The Python documentation that you cite says incorrectly that the user can control the orientation of the JSON output. In fact, that option is not currently read and processed, and the Python output always has the same orientation.

We agree that the different orientations of the Java and Python JSON output is undesirable, although what to do about it at this point is less clear. Part of our reasoning is that because the JSON output is intended for post-processing, and not as a final-form visualization, it is more important to provide a single, consistent, well-documented format than to support end-user tuning of the orientation of the JSON output. We recognize that at the moment we don't have a single, consistent, well-documented format (sigh!), but that is nonetheless a goal. We are currently developing a new implementation, in Scala, which is intended as a drop-in replacement for the Java version (both local execution and web service), which means that we need to think about how that version should behave.

Current thoughts are:

1. We should update the Python documentation to remove the incorrect statement that the user can control the orientation of the JSON output. As you write, that orientation can be managed instead in post-processing, so our goal is to fix the documentation, rather than to extend the functionality.
2. We are inclined to leave the discrepant Java vs Python orientations as is because changing one or the other now might break someone's working code. We agree that the discrepancy is unfortunate, but now that the two versions have been coexisting for about a decade, we think leaving the discrepancy in place is likely to be least disruptive for users.
3. The Scala implementation in progress will produce JSON output that matches the current Java (not Python) output. Our goal here is to allow the Scala implementation to serve as a drop-in replacement for the Java one, which means that we want to avoid changing the behavior.
4. We will update the documentation of the Scala version, when it is ready for release, to clarify the format of the JSON output. Our current plan is to add a documentation section with a title like "The CollateX implementation of the Gothenburg protocol", using the term "protocol" to mean that includes both the model (a pipeline of five steps) and an API for the alignment step that is intended to be interoperable across implementations.

Please let us know here whether the preceding sounds useful to you as an end-user, and we would welcome any other suggestions you might have.

[tarrinw]

I agree with all the suggestions here.

I would only add that continuing the development of the Python version makes much more sense than migrating to yet another language/platform (Scala), and one that is much less widely supported in the dev world than the existing java and especially python implementations. But as the decision already seems to have been made, I would rather that development continues in any language.

Thanks for your support on this very useful tool. I have been using it successfully for its intended purpose of collating versions of texts and have developed both editing and a phylogenetic analysis pipelines (see menotag.ku.dk) based on CollateX. I am now experimenting in aligning manual and HTR transcriptions of mss using the tool, which is showing some promise.

[djbpitt]

@tarrinw You may already know that Scala source code can be compiled into a jar, so end-users of the Java version won't see any difference, on the level of implementation language, between the pure Java code they're been using and the Scala code that will eventually replace it. We moved from Java to Scala partially because of that backward compatibility, that is, to avoid breaking anything that Java users might have come to rely on. Scala wasn't available when the Java development began many years ago, but once Scala became available, we found it easier to work with than Java with because it had more natural support for functional idioms.

We ported CollateX to Python in the hope that doing so would attract more users within the DH community, where, as your message suggests, the level of user knowledge of and comfort with Python is greater than is the case with Java. When we began implementing a new alignment algorithm several years ago we began that development in Python, but we found that the lack of strong typing in Python made it a fragile and frustrating development platform. Once the Scala version is finished, we are interested in porting the new alignment algorithm to Python and integrating it into the Python version, since porting to Python is easier (less error-prone, less dependent on robust support from developer tools) than developing initially in Python would be. That won't happen soon because there is still work to do on developing our implementation of that new alignment algorithm, but moving it eventually into the Python port is part of the roadmap. It is too early to evaluate the new alignment algorithm strictly, but so far the quality of the alignment seems comparable to what we find in the current release versions, and the performance with large input seems much improved.

We appreciate your kind words about CollateX, and we are happy to see that it has been of value to the Menota project.