feat: transactional UpdateProperties method support #321

lishuxu · 2025-11-15T03:35:45Z

feat: transactional UpdateProperties method support

Table/Catalog produce a Transaction (e.g., NewTransaction or staged create/replace).
Clients build PendingUpdate actions (e.g., UpdateProperties, AppendFiles), each validating via Apply() (conflict checks, reserved keys, format-version parsing, schema/metrics validation).
Transaction::CommitTransaction() is expected to gather pending updates and call the catalog’s UpdateTable (with derived TableUpdate objects and requirements) in one atomic commit; on success, metadata refreshes; on failure, errors propagate

zhjwpku

This is consistent with the rest catalog open api, so +1, thanks.

src/iceberg/transaction.h

wgtmac · 2025-11-17T14:51:55Z

src/iceberg/transaction.h

+  /// \param schema_ids the schema ids to remove
+  /// \return a new RemoveSchemas
+  virtual std::shared_ptr<RemoveSchemas> RemoveSchemas(
+      const std::vector<int32_t>& schema_ids) = 0;


Suggested change

const std::vector<int32_t>& schema_ids) = 0;

std::span<const int32_t> schema_ids) = 0;

Should we use span to accept broader range of input types for here and below?

src/iceberg/type_fwd.h

wgtmac · 2025-12-03T06:38:28Z

Could you rebase to resolve conflict?

src/iceberg/catalog/memory/in_memory_catalog.h

lishuxu · 2025-12-08T02:07:08Z

Could you rebase to resolve conflict?

done

wgtmac · 2025-12-09T12:23:15Z

src/iceberg/catalog.h

      const TableIdentifier& identifier,
-      const std::vector<std::unique_ptr<TableRequirement>>& requirements,
-      const std::vector<std::unique_ptr<TableUpdate>>& updates) = 0;
+      std::vector<std::unique_ptr<TableRequirement>> requirements,


Can we revert this change? If later we support retry on failed commits, these requirements and updates cannot be reused since they are moved away.

wgtmac · 2025-12-09T12:23:53Z

src/iceberg/catalog.h

+  /// \brief Set whether the last operation in a transaction has been committed
+  ///
+  /// \param committed true if the last operation has been committed, false otherwise
+  virtual void SetLastOperationCommitted(bool committed) = 0;


This function is only used by transaction catalog so it should not appear here.

wgtmac · 2025-12-09T12:28:09Z

src/iceberg/table.cc


 std::unique_ptr<Transaction> Table::NewTransaction() const {
-  throw NotImplemented("Table::NewTransaction is not implemented");
+  return std::make_unique<BaseTransaction>(shared_from_this(), catalog_);


I'd recommend directly pass Table* to it. Similarly, for catalog_ we can use catalog_.get() to just use pointer. The created transaction object should not outlive the table and catalog objects. It is unnecessary to increase the reference counter of them.

I'm open to keep the current design. WDYT @HuaHuaY?

When the table‘s content changes, or we drop a table, what should the executing transaction hold?

wgtmac · 2025-12-09T13:12:25Z

src/iceberg/base_transaction.h

+  std::unique_ptr<::iceberg::UpdateProperties> UpdateProperties() override;
+
+  std::unique_ptr<AppendFiles> NewAppend() override;


Suggested change

std::unique_ptr<::iceberg::UpdateProperties> UpdateProperties() override;

std::unique_ptr<AppendFiles> NewAppend() override;

std::shared_ptr<UpdateProperties> NewUpdateProperties() override;

std::shared_ptr<AppendFiles> NewAppend() override;

It seems that returning shared_ptr will make it easier to collect and transform the update internally. Adding New prefix also can avoid iceberg:: prefix in the return type.

wgtmac · 2025-12-09T13:29:03Z

src/iceberg/base_transaction.h

+namespace iceberg {
+
+/// \brief Base class for transaction implementations
+class ICEBERG_EXPORT BaseTransaction : public Transaction {


If BaseTransaction is the only subclass, we can just implement all features to Transaction just like Table.

BTW, do we want to introduce TransactionType from the Java impl as well? If would be good if we can add this to make it clear.

wgtmac · 2025-12-09T13:43:37Z

src/iceberg/base_transaction.cc

+  auto update = CheckAndCreateUpdate<::iceberg::UpdateProperties>(
+      table_->name(), catalog_, CurrentMetadata());
+  if (!update.has_value()) {
+    ERROR_TO_EXCEPTION(update.error());


We cannot throw. Perhaps we should also use Result wrapper for these update return types.

wgtmac · 2025-12-09T13:46:10Z

src/iceberg/transaction_catalog.h

+ * For mutating calls such as UpdateTable or HasLastOperationCommitted, it delegates back
+ * to the owning BaseTransaction so staged updates remain private until commit.
+ */
+class ICEBERG_EXPORT TransactionCatalog : public Catalog {


Can we move this class to base_transaction.cc?

wgtmac · 2025-12-09T14:30:20Z

src/iceberg/base_transaction.cc

+
+  // update table to the new version
+  if (updated_table) {
+    table_ = std::shared_ptr<Table>(std::move(updated_table));


Why do we need this table_ after commit has been done? It seems useless?

wgtmac · 2025-12-09T14:31:24Z

src/iceberg/base_transaction.cc

+    context_.pending_updates.emplace_back(std::move(update));
+  }
+
+  return std::make_unique<Table>(


Why do we need to return a table?

wgtmac · 2025-12-09T14:40:17Z

src/iceberg/base_transaction.cc

+    return {};
+  }
+
+  auto pending_requirements = ConsumePendingRequirements();


What if we need to support retry on failed commits? In the current approach, these two lists are gone.

zhjwpku approved these changes Nov 16, 2025

View reviewed changes

wgtmac reviewed Nov 17, 2025

View reviewed changes

Jinchul81 reviewed Nov 22, 2025

View reviewed changes

src/iceberg/type_fwd.h Outdated Show resolved Hide resolved

HeartLinked approved these changes Nov 25, 2025

View reviewed changes

lishuxu force-pushed the feature/transaction-api branch from 23bf661 to 0c1f156 Compare November 29, 2025 13:50

lishuxu changed the title ~~feat: add table-update api to transaction interface~~ feat: transactional UpdateProperties method support Nov 29, 2025

shuxu.li added 6 commits December 4, 2025 14:41

feat: add table-update api to transaction interface

96df225

feat: transactional UpdateProperties support

7475e89

feat: transactional UpdateProperties method support

b2e75a8

feat: transactional UpdateProperties method support

b069251

feat: transactional UpdateProperties method support

4159c75

feat: table transactional UpdateProperties support

14bd4f8

lishuxu force-pushed the feature/transaction-api branch from aa4cadb to 14bd4f8 Compare December 6, 2025 13:38

shuxu.li added 4 commits December 6, 2025 22:26

feat: transactional UpdateProperties method support

b9ea8fe

feat: transactional UpdateProperties method support

451f7f2

feat: transactional UpdateProperties method support

37775fe

feat: transactional UpdateProperties method support

bffcd4e

HuaHuaY reviewed Dec 7, 2025

View reviewed changes

src/iceberg/catalog/memory/in_memory_catalog.h Outdated Show resolved Hide resolved

feat: transactional UpdateProperties method support

fe43102

shuxu.li added 3 commits December 8, 2025 11:38

feat: transactional UpdateProperties method support

5af9f71

feat: add update table properties support

a96aae4

feat: transactional UpdateProperties method support

4844d87

wgtmac reviewed Dec 9, 2025

View reviewed changes

	const std::vector<int32_t>& schema_ids) = 0;
	std::span<const int32_t> schema_ids) = 0;

		std::unique_ptr<::iceberg::UpdateProperties> UpdateProperties() override;

		std::unique_ptr<AppendFiles> NewAppend() override;

feat: transactional UpdateProperties method support #321

Are you sure you want to change the base?

feat: transactional UpdateProperties method support #321

Uh oh!

Conversation

lishuxu commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhjwpku left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wgtmac commented Dec 3, 2025

Uh oh!

Uh oh!

lishuxu commented Dec 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lishuxu commented Nov 15, 2025 •

edited

Loading

HuaHuaY Dec 9, 2025 •

edited

Loading