A stale update_sort_order adds a second sort order with an existing id, which Java cannot load #4202

Apache Iceberg version

0.12.0 (latest release)

Please describe the bug 🐞

Currently, UpdateSortOrder commits with only AssertDefaultSortOrderId, and applying AddSortOrderUpdate appends an order whose id already exists when the two orders are equal. So after another writer sets a sort order and then resets the table to unsorted, a stale handle setting the same order passes the check and the metadata lists sort order 1 twice. Java's TableMetadataParser then refuses the table with IllegalArgumentException: Multiple entries with same key.

import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.transforms import IdentityTransform

cat = SqlCatalog("c", uri="sqlite:///:memory:", warehouse=f"file://{tempfile.mkdtemp()}")
cat.create_namespace("ns")
t = cat.create_table("ns.t", schema=pa.schema([("x", pa.int64())]))

other = cat.load_table("ns.t")
with other.update_sort_order() as u:
    u.asc("x", IdentityTransform())
with other.update_sort_order() as u:  # back to unsorted
    pass

with t.update_sort_order() as u:
    u.asc("x", IdentityTransform())

print([o.order_id for o in cat.load_table("ns.t").metadata.sort_orders])
# [0, 1, 1], should be [0, 1]

Sorting the stale handle by a different column raises a non-retryable ValueError: Sort-order with id 1 already exists instead. Java's TableMetadata.Builder.addSortOrder reuses the id of an equal existing order.

tested on pyiceberg 0.12.0 and main

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time