Added ExpireSnapshots Feature #1880

ForeverAngry · 2025-04-03T19:18:23Z

Summary

This PR Closes issue #516 by implementing support for the ExpireSnapshot table metadata action.

Rationale

The ExpireSnapshot action is a core part of Iceberg’s table maintenance APIs. Adding support for this action in PyIceberg helps ensure feature parity with other language implementations (e.g., Java) and supports users who want to programmatically manage snapshot retention using PyIceberg’s public API.

Testing

Unit tests have been added to cover the initial expected usage paths.
Additional feedback on edge cases, missing test scenarios or corrections to the setup test logic is greatly welcome during the review process.

User-facing changes

This change introduces a new public API: ExpireSnapshot.
No breaking changes or modifications to existing APIs were made.

…p data

…h a new Expired Snapshot class. updated tests.

ValueError: Cannot expire snapshot IDs {3051729675574597004} as they are currently referenced by table refs.

Fokko

Thanks @ForeverAngry for raising this PR! I'll go into details tomorrow morning (UTC+2 here). Could you resolve the conflicts to get the CI running?

pyproject.toml

pyiceberg/table/update/snapshot.py

ForeverAngry

Applied intial suggestions so that CI can run on the PR.

ForeverAngry

rebuilt the poetry lock file.

Moved expiration-related methods from `ExpireSnapshots` to `ManageSnapshots` for improved organization and clarity. Updated corresponding pytest tests to reflect these changes.

Re-ran the `poetry run pre-commit run --all-files` command on the project.

ForeverAngry · 2025-04-13T06:17:33Z

After looking at the way the action here was implemented, I refined the changes. Let me know if these make sense :)

ForeverAngry

@kevinjqliu thoughts?

ForeverAngry

Reviewed.

ForeverAngry

rebased from main

kevinjqliu

I think there's a merge conflict somewhere, there are a lot of code here from other PRs. Perhaps you need to update your fork's main branch.

.gitignore

tests/expressions/test_literals.py

tests/integration/test_rest_schema.py

pyiceberg/table/update/snapshot.py

tests/table/test_expire_snapshots.py

ForeverAngry · 2025-05-17T18:48:10Z

@ForeverAngry Sorry for the late reply, it looks like that there is a test failing now 👀

I think this commit should fix the test error, i also added additional tests. All passed - and appear to be in-good-order. 🤞 this time is the charm.

Updated test_expire_snapshots.py to use model_copy(update={...}) for modifying the refs attribute of the table metadata, ensuring compatibility with Pydantic's frozen models. Fixed all snapshot expiration tests to avoid direct assignment to frozen attributes. All tests now pass, confirming correct behavior for protected and unprotected snapshot expiration.

zschumacher · 2025-06-11T01:41:31Z

this would be great - whats the status of this?

smaheshwar-pltr · 2025-06-11T10:49:41Z

I see that in #1958, for orphaned file removal, we decided to have a table.maintenance API returning a MaintenanceTable. As a user, if I have to do orphaned file removal via that table.maintenance API, but snapshot expiration instead via table.expire_snapshots, I might be confused (they both feel like table maintenance to me, though snapshot expiration is admittedly a bit stronger). Curious about people's thoughts here.

ForeverAngry · 2025-06-11T16:23:01Z

Well, right now this pr doesn't do anything with the newly orphaned files. It just handles the metadata operation.

pyiceberg/table/update/snapshot.py

pyiceberg/table/__init__.py

Fokko · 2025-06-11T18:04:58Z

I see that in #1958, for orphaned file removal, we decided to have a table.maintenance API returning a MaintenanceTable. As a user, if I have to do orphaned file removal via that table.maintenance API, but snapshot expiration instead via table.expire_snapshots, I might be confused (they both feel like table maintenance to me, though snapshot expiration is admittedly a bit stronger). Curious about people's thoughts here.

Yes, I agree with you there. Before doing the 0.10.0 release, we need to ensure we align on this and make proper docs. I have a slight preference towards .mainenance to have a clear distinction between maintenance and the regular operations (such as creating a tag or branch).

ForeverAngry · 2025-06-11T21:44:28Z

I'm happy to follow the '.maintenance' api design if there is a strong preference toward it.

Co-authored-by: Fokko Driesprong <[email protected]>

Added ExpireSnapshots.expire_snapshots_older_than and expire_snapshots_by_ids methods to support expiring snapshots by timestamp and by multiple IDs. Ensured that protected snapshots (branch/tag heads) cannot be expired, both at the API and commit stages. Updated the expiration logic to always skip protected snapshots, even if they are accidentally included. Added and fixed tests to verify that protected snapshots are never expired and that expiration works as expected for unprotected snapshots. Improved test setup to accurately reflect post-commit metadata and to assert correct expiration behavior.

Fokko

Thanks @ForeverAngry for working on this, and I think it is ready to go 👍

ForeverAngry · 2025-06-19T16:13:58Z

Thanks @ForeverAngry for working on this, and I think it is ready to go 👍

Great! I think @kevinjqliu is still listed as needing approval. @kevinjqliu can you put your stamp on this as well?

Fokko · 2025-06-19T19:49:42Z

@ForeverAngry I think we can move this one forward. Before the release, we need to follow up on two things:

Add a new Maintenance doc section with a subsection that explains the expire snapshots operation.
Move the expire snapshots operation under maintenance: tbl.maintenance.expire_snapshots()

Fokko · 2025-06-19T19:50:11Z

Thanks again @ForeverAngry for working on this 🚀

ForeverAngry · 2025-06-19T22:02:40Z

Thanks again @ForeverAngry for working on this 🚀

Thank you, for being such a supportive and inspiring member to work with!

ForeverAngry · 2025-06-19T22:04:19Z

@ForeverAngry I think we can move this one forward. Before the release, we need to follow up on two things:

Add a new Maintenance doc section with a subsection that explains the expire snapshots operation.

Move the expire snapshots operation under maintenance: tbl.maintenance.expire_snapshots()

I'll work on this, this weekend!

Fokko · 2025-06-20T21:48:40Z

@ForeverAngry Appreciate that, thanks! 🙌

greenlaw · 2025-06-24T19:09:52Z

@ForeverAngry Thank you for this feature ❤️

Just one question/comment: It seems this only supports expiration time/age, and does not support other retention policies. For example, the Java API's ExpireSnapshots supports retainLast, and ManageSnapshots supports setMinSnapshotsToKeep. Any plans to add support for these features, by chance?

ForeverAngry · 2025-06-24T20:53:14Z

@ForeverAngry Thank you for this feature ❤️

Just one question/comment: It seems this only supports expiration time/age, and does not support other retention policies. For example, the Java API's ExpireSnapshots supports retainLast, and ManageSnapshots supports setMinSnapshotsToKeep. Any plans to add support for these features, by chance?

Yeah, those slipped my mind when I originally did it. I'd be happy to implement those. :)

## Summary This PR Closes issue apache#516 by implementing support for the `ExpireSnapshot` table metadata action. ## Rationale The `ExpireSnapshot` action is a core part of Iceberg’s table maintenance APIs. Adding support for this action in PyIceberg helps ensure feature parity with other language implementations (e.g., Java) and supports users who want to programmatically manage snapshot retention using PyIceberg’s public API. ## Testing - Unit tests have been added to cover the initial expected usage paths. - Additional feedback on edge cases, missing test scenarios or corrections to the setup test logic is greatly welcome during the review process. ## User-facing changes - This change introduces a new public API: `ExpireSnapshot`. - No breaking changes or modifications to existing APIs were made. --- --------- Co-authored-by: Fokko Driesprong <[email protected]>

@jayceslesar

# Rationale for this change - Consolidates snapshot expiration functionality from the standalone `ExpireSnapshots` class into the `MaintenanceTable` class for a unified maintenance API. - Resolves planned work left over from #1880, and closes #2142 - Achieves feature and API parity with the Java implementation for snapshot retention and table maintenance. # Features & Enhancements - Introduces `table.maintenance.expire_snapshots()` as the unified entry point for snapshot expiration and future maintenance operations. - Retains the existing `ExpireSnapshots` implementation internally. The `expire_snapshots()` method on `MaintenanceTable` now returns an `ExpireSnapshots` object, preserving transaction semantics and supporting context manager usage: ```python with table.maintenance.expire_snapshots() as expire_snapshots: expire_snapshots.by_id(1) expire_snapshots.by_id(2) ``` - Focuses this PR on refactoring and documentation improvements, while maintaining compatibility with the prior `ExpireSnapshots` interface. - Sets a foundation for future expansion of the `MaintenanceTable` abstraction to encapsulate additional maintenance operations. # Bug Fixes & Cleanups - **ManageSnapshots Cleanup ([#2151](#2151 - Removes an unrelated instance variable from the `ManageSnapshots` class, aligning with the Java reference implementation. # Testing & Documentation - **Testing:** - Tested the new API interface including: - Expiration by ID - Protection of branch/tag snapshots - **Documentation:** - Added and updated documentation to describe: - API usage examples Preview: <img width="1686" height="1015" alt="Screenshot 2025-08-11 at 1 37 04 PM" src="https://github.com/user-attachments/assets/f469f3fc-b4b1-4ec9-b1ca-b9185e22643e" /> # Are these changes tested? Yes. All changes are tested.~, with this PR predicated on the final changes from #1200.~ This work builds on the framework introduced by @jayceslesar in #1200 for the `MaintenanceTable`. # Are there any user-facing changes? --- **Closes:** - Closes #2151 - Closes #2142 --------- Co-authored-by: Fokko Driesprong <[email protected]> Co-authored-by: Kevin Liu <[email protected]>

## Summary This PR Closes issue apache#516 by implementing support for the `ExpireSnapshot` table metadata action. ## Rationale The `ExpireSnapshot` action is a core part of Iceberg’s table maintenance APIs. Adding support for this action in PyIceberg helps ensure feature parity with other language implementations (e.g., Java) and supports users who want to programmatically manage snapshot retention using PyIceberg’s public API. ## Testing - Unit tests have been added to cover the initial expected usage paths. - Additional feedback on edge cases, missing test scenarios or corrections to the setup test logic is greatly welcome during the review process. ## User-facing changes - This change introduces a new public API: `ExpireSnapshot`. - No breaking changes or modifications to existing APIs were made. --- --------- Co-authored-by: Fokko Driesprong <[email protected]>

@jayceslesar

…he#2143)   # Rationale for this change - Consolidates snapshot expiration functionality from the standalone `ExpireSnapshots` class into the `MaintenanceTable` class for a unified maintenance API. - Resolves planned work left over from apache#1880, and closes apache#2142 - Achieves feature and API parity with the Java implementation for snapshot retention and table maintenance. # Features & Enhancements - Introduces `table.maintenance.expire_snapshots()` as the unified entry point for snapshot expiration and future maintenance operations. - Retains the existing `ExpireSnapshots` implementation internally. The `expire_snapshots()` method on `MaintenanceTable` now returns an `ExpireSnapshots` object, preserving transaction semantics and supporting context manager usage: ```python with table.maintenance.expire_snapshots() as expire_snapshots: expire_snapshots.by_id(1) expire_snapshots.by_id(2) ``` - Focuses this PR on refactoring and documentation improvements, while maintaining compatibility with the prior `ExpireSnapshots` interface. - Sets a foundation for future expansion of the `MaintenanceTable` abstraction to encapsulate additional maintenance operations. # Bug Fixes & Cleanups - **ManageSnapshots Cleanup ([apache#2151](apache#2151 - Removes an unrelated instance variable from the `ManageSnapshots` class, aligning with the Java reference implementation. # Testing & Documentation - **Testing:** - Tested the new API interface including: - Expiration by ID - Protection of branch/tag snapshots - **Documentation:** - Added and updated documentation to describe: - API usage examples Preview: <img width="1686" height="1015" alt="Screenshot 2025-08-11 at 1 37 04 PM" src="https://github.com/user-attachments/assets/f469f3fc-b4b1-4ec9-b1ca-b9185e22643e" /> # Are these changes tested? Yes. All changes are tested.~, with this PR predicated on the final changes from apache#1200.~ This work builds on the framework introduced by @jayceslesar in apache#1200 for the `MaintenanceTable`. # Are there any user-facing changes? --- **Closes:** - Closes apache#2151 - Closes apache#2142 --------- Co-authored-by: Fokko Driesprong <[email protected]> Co-authored-by: Kevin Liu <[email protected]>

ForeverAngry added 8 commits March 28, 2025 20:23

Added initial units tests and Class for Removing a Snapshot

0a94d96

Added methods needed to expire snapshots by id, and optionally cleanu…

5f0b62b

…p data

Update test_expire_snapshots.py

f995daa

Added the builder method to __init__.py, updated the snapshot api wit…

65365e1

…h a new Expired Snapshot class. updated tests.

Snapshots are not being transacted on, but need to re-assign refs

e28815f

ValueError: Cannot expire snapshot IDs {3051729675574597004} as they are currently referenced by table refs.

Fixed the test case.

4628ede

adding print statements to help with debugging

e80c41c

Draft ready

cb9f0c9

Fokko reviewed Apr 3, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

pyiceberg/table/update/snapshot.py Outdated Show resolved Hide resolved

pyiceberg/table/update/snapshot.py Outdated Show resolved Hide resolved

ForeverAngry commented Apr 3, 2025

View reviewed changes

ForeverAngry added 3 commits April 3, 2025 15:56

Applied suggestions to Fix CICD

ebcff2b

Merge branch 'main' into main

97399bf

Rebuild the poetry lock file.

95e5af2

ForeverAngry commented Apr 3, 2025

View reviewed changes

Merge branch 'main' into main

5ab5890

ForeverAngry requested a review from Fokko April 4, 2025 20:36

ForeverAngry added 5 commits April 13, 2025 01:53

Refactor implementation of ExpireSnapshots

5acd690

Moved expiration-related methods from `ExpireSnapshots` to `ManageSnapshots` for improved organization and clarity. Updated corresponding pytest tests to reflect these changes.

Fixed format and linting issues

d30a08c

Re-ran the `poetry run pre-commit run --all-files` command on the project.

Merge branch 'main' into main

e62ab58

Fixed format and linting issues

1af3258

Re-ran the `poetry run pre-commit run --all-files` command on the project.

Merge branch 'main' of https://github.com/ForeverAngry/iceberg-python

352b48f

ForeverAngry commented Apr 16, 2025

View reviewed changes

Merge branch 'main' into main

382e0ea

ForeverAngry commented Apr 18, 2025

View reviewed changes

rebased: from main

549c183

ForeverAngry commented Apr 19, 2025

View reviewed changes

fixed: typo

386cb15

kevinjqliu reviewed Apr 21, 2025

View reviewed changes

removed errant files

12729fa

Fokko reviewed Jun 11, 2025

View reviewed changes

pyiceberg/table/update/snapshot.py Outdated Show resolved Hide resolved

Fokko reviewed Jun 11, 2025

View reviewed changes

pyiceberg/table/__init__.py Outdated Show resolved Hide resolved

ForeverAngry and others added 3 commits June 13, 2025 21:08

Update pyiceberg/table/__init__.py

4a1b9a3

Co-authored-by: Fokko Driesprong <[email protected]>

Removed unused instance variable.

1cac992

Fokko approved these changes Jun 19, 2025

View reviewed changes

Fokko merged commit 1bdc24e into apache:main Jun 19, 2025
10 checks passed

This was referenced Jun 23, 2025

refactor: consolidate snapshot expiration into MaintenanceTable #2142

Closed

refactor: consolidate snapshot expiration into MaintenanceTable #2143

Merged

rambleraptor mentioned this pull request Jun 25, 2025

Support Snapshot Expiration Operation #516

Closed

ForeverAngry deleted the 1880-add-expire-snapshots branch June 25, 2025 22:31

Added ExpireSnapshots Feature #1880

Added ExpireSnapshots Feature #1880

Uh oh!

Conversation

ForeverAngry commented Apr 3, 2025

Summary

Rationale

Testing

User-facing changes

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ForeverAngry left a comment

Choose a reason for hiding this comment

Uh oh!

ForeverAngry left a comment

Choose a reason for hiding this comment

Uh oh!

ForeverAngry commented Apr 13, 2025

Uh oh!

ForeverAngry left a comment

Choose a reason for hiding this comment

Uh oh!

ForeverAngry left a comment

Choose a reason for hiding this comment

Uh oh!

ForeverAngry left a comment

Choose a reason for hiding this comment

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ForeverAngry commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zschumacher commented Jun 11, 2025

Uh oh!

smaheshwar-pltr commented Jun 11, 2025

Uh oh!

ForeverAngry commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

Fokko commented Jun 11, 2025

Uh oh!

ForeverAngry commented Jun 11, 2025

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

ForeverAngry commented Jun 19, 2025

Uh oh!

Fokko commented Jun 19, 2025

Uh oh!

Uh oh!

Fokko commented Jun 19, 2025

Uh oh!

ForeverAngry commented Jun 19, 2025

Uh oh!

ForeverAngry commented Jun 19, 2025

Uh oh!

Fokko commented Jun 20, 2025

Uh oh!

greenlaw commented Jun 24, 2025

Uh oh!

ForeverAngry commented Jun 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

ForeverAngry commented May 17, 2025 •

edited

Loading