Introduce WG Checkpoint Restore #8508

adrianreber · 2025-07-03T13:33:14Z

As described in sig-wg-lifecycle.md this PR is the next step after sending an email to [email protected] about the creation of the Working Group Checkpoint Restore.

CC: @rst0git, @viktoriaas, @xhejtman

k8s-ci-robot · 2025-07-03T13:33:24Z

Welcome @adrianreber!

It looks like this is your first PR to kubernetes/community 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/community has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-07-03T13:33:24Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: adrianreber
Once this PR has been reviewed and has the lgtm label, please assign saschagrunert for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-07-03T13:33:25Z

Hi @adrianreber. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

OWNERS_ALIASES

kannon92 · 2025-07-10T18:31:17Z

/ok-to-test

kannon92 · 2025-07-15T02:42:37Z

Looking at #8519,

I see that we are missing a charter.

adrianreber · 2025-07-15T06:34:13Z

Looking at #8519,

I see that we are missing a charter.

In https://github.com/kubernetes/community/blob/master/sig-wg-lifecycle.md#GitHub is says to add a charter once this initial PR has been merged. That's why is skipped it.

SergeyKanzhelev · 2025-07-18T17:40:53Z

sigs.yaml

+    the integration of Checkpoint/Restore functionality into Kubernetes.
+
+  charter_link: charter.md
+  stakeholder_sigs:


sig auth may have a big say in security of this whole restoration pipeline

Thank you for pointing this out! Security is definitely an important topic that we need to discuss with sig-auth, both for the checkpoint API and the restoration pipeline. The following paper and master thesis describe our recent work on this topic:

Towards Efficient End-to-End Encryption for Container Checkpointing Systems

Improving Checkpoint/Restore Functionality in Kubernetes

I added sig auth to the list of stakeholder sigs

this showed up in the sig-auth meeting, we may have missed the discussion around this WG

if this WG is contemplating taking state from a running pod / saving it / letting it be consumed on another node or from another pod or another namespace, then sig-auth is definitely interested in making sure the permissions model around that exists and is ~consistent with similar things Kubernetes does elsewhere (like PVC / snapshots)

We're happy to consult on that, I'm not sure our awareness / involvement rises to the level of sponsoring the WG :)

cc @kubernetes/sig-auth-leads

nod.. definitely needs an extra level of security due to customer data being serialized and available in the checkpoint, esp if not encrypted, but also due to windows of opportunity to do transactions/data manipulation.. then "undo" them by restoring a checkpoint

sigs.yaml

aramase · 2025-08-04T16:28:16Z

/assign ritazh

(assigned as part of SIG Auth triage; to review the SIG Auth updates)

wg-checkpoint-restore/charter.md

BenTheElder · 2025-08-20T00:11:53Z

@kubernetes/sig-node-leads are you all +1, officially?

haircommander · 2025-08-20T13:16:04Z

+1 from me

sigs.yaml

aojea · 2025-08-21T08:26:41Z

wg-checkpoint-restore/charter.md

+
+- maintain a solid communication line between the Kubernetes groups and the
+  wider CNCF community
+- submit a proposal to the KubeCon/CloudNativeCon maintainers track


I have doubts if this incentivize the right behavior and will encourage people to build WG to get a slot in the kubecon

I agree with Antonio, this particular line should be removed, it's sufficient what the previous point shows.

This was just copied from 0743da8, but I will remove it.

wg-checkpoint-restore/charter.md

wg-checkpoint-restore/README.md

sig-cli/README.md

soltysh · 2025-08-27T12:03:07Z

sigs.yaml

+    the integration of Checkpoint/Restore functionality into Kubernetes.
+
+  charter_link: charter.md
+  stakeholder_sigs:


I agree with Janet here, but please make sure to show up and present the scope of this proposal to one of the future SIG-Apps calls.

sigs.yaml

wg-checkpoint-restore/charter.md

soltysh · 2025-08-27T12:08:45Z

wg-checkpoint-restore/charter.md

+The Checkpoint/Restore Working Group aims to solve the problem of transparently
+checkpointing and restoring workloads in Kubernetes, a functionality discussed
+for over five years. The group will deliver the design and implementation of
+Checkpoint/Restore functionality in Kubernetes, serving as a central hub for


Why it has to be a central part of Kubernetes, where multiple external solutions already exists?

i personally like the idea of having it be integrated because then the ecosystem can rely on it. for instance, we could make eviction or preemption less disruptive in kubelet/kueue respectively

Why it has to be a central part of Kubernetes, where multiple external solutions already exists?

Can you be more specific about what already exists? Not sure what you are referring to?

soltysh · 2025-08-27T12:11:22Z

wg-checkpoint-restore/charter.md

+Checkpoint/Restore functionality in Kubernetes, serving as a central hub for
+community information and discussion. This initiative addresses a wide range of
+problems, including fault tolerance, improved resource utilization, and
+accelerated application startup times.


This first thing that I'd like to point out is that there are 2 main use cases:

the whole control-plane snapshot

workload

Which one this group is planning to cover? As I'm reading this document I'm seeing both used interchangeably which is very confusing. That's why I'd start with clearly drawing the line between the two and properly documenting which one of these two (or both) are you planning to tackle.

What do yo mean by "control-plane-snapshot"?

In our presentations, we use the following diagram to illustrate how checkpoint/restore operations work in Kubernetes:

Reference: Efficient Transparent Checkpointing of AI/ML Workloads in Kubernetes

soltysh · 2025-08-27T12:19:01Z

wg-checkpoint-restore/charter.md

+
+- Ability to checkpoint and restore a container using kubectl
+- Ability to checkpoint and restore a pod using kubectl
+- Integration of container/pod checkpointing in scheduling decisions


Why pod checkpointing would have anything to do with scheduling?

Because, as far as I know, a pod is always scheduled on one node. It doesn't sound useful to base the scheduling on the possibility to migrate containers. Container migration is an important first step, but for automatic scheduling decisions, it would make more sense to be able to easily migrate a complete pod.

Our use-case is similar to how CRIU is integrated with Google's Borg ¹ and Microsoft's Singularity ² to enable preemptive and elastic scheduling.

Footnotes

Task Migration at Scale Using CRIU ↩

Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads ↩

Why pod checkpointing would have anything to do with scheduling?

I agree that pod checkpointing needs to take scheduling into consideration. Let's assume the following scenario: after a Pod is checkpointed, it might not be deleted. However, since the memory, CPU, and GPU resources have already been dumped to a volume, the resources originally allocated to the Pod could then be reallocated to other Pods. When the Pod needs to be resumed later via restore, the scheduler should also be involved.

soltysh · 2025-08-27T12:20:09Z

wg-checkpoint-restore/charter.md

+
+- maintain a solid communication line between the Kubernetes groups and the
+  wider CNCF community
+- submit a proposal to the KubeCon/CloudNativeCon maintainers track


I agree with Antonio, this particular line should be removed, it's sufficient what the previous point shows.

Co-authored-by: Viktória Spišaková <[email protected]> Co-authored-by: Antonio Ojea <[email protected]> Co-authored-by: Sergey Kanzhelev <[email protected]> Signed-off-by: Adrian Reber <[email protected]>

aojea · 2025-09-09T11:11:35Z

SIG Network questions (cc @danwinship @thockin )

does checkpoint restore move the network state ? as the established TCP connections? if affirmative, it requires IP mobility or uses Services or any other abstraction?

adrianreber · 2025-09-09T11:17:04Z

SIG Network questions (cc @danwinship @thockin )

does checkpoint restore move the network state ? as the established TCP connections? if affirmative, it requires IP mobility or uses Services or any other abstraction?

Migrating a container with established TCP connections is possible, but only if the restored container has the same IP as the checkpointed container. In Podman we were able to implement but I am not sure if this is something easily doable in Kubernetes. From our side we would need the possibility to create a pod with a certain IP and then it would work. Is there something in Kuberentes which would allow pod creation with a given IP address?

aojea · 2025-09-09T11:33:25Z

. Is there something in Kuberentes which would allow pod creation with a given IP address?

For Pod IPs, Is an IPAM choice of the CNI plugin, kubernetes network model allows that, but implementing it is more complex since most implementations have locality implied for the assigned Pod IPs, and to allow IP mobility it requires to deal with routing and ipam across the entire cluster to keep the IP connectivity that adds a considerable complexity.

Another thing is if you just expose the Pod application as a Service or Gateway API or some high level abstraction that have VirtualIP, then it is simpler, since those are already abstraction with cluster scope ... This works for Ingress traffic to the application, for Egress traffic is also more complex

k8s-ci-robot · 2025-09-09T20:21:21Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lujinda · 2025-09-14T14:01:32Z

Is there a slack channel where we can discuss C/R related ideas? Thanks

adrianreber · 2025-09-14T14:04:38Z

Is there a slack channel where we can discuss C/R related ideas? Thanks

You are not the first to ask. We kind of are waiting for the proposal to be accepted to have a slack channel. Not sure if there is a another way to have a slack channel without having the proposal merged.

rst0git · 2025-09-15T10:25:33Z

Is there a slack channel where we can discuss C/R related ideas?

@lujinda Please reach out to us in the Kubernetes slack. You can find Viktoria, Adrian, and myself there :)

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 3, 2025

k8s-ci-robot requested review from deads2k and macsko July 3, 2025 13:33

k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. labels Jul 3, 2025

github-project-automation bot added this to SIG Scheduling Jul 3, 2025

github-project-automation bot moved this to Needs Triage in SIG Scheduling Jul 3, 2025

k8s-ci-robot added the do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. label Jul 3, 2025

adrianreber mentioned this pull request Jul 3, 2025

Checkpointing API kubernetes/enhancements#5091

Open

6 tasks

haircommander reviewed Jul 8, 2025

View reviewed changes

OWNERS_ALIASES Show resolved Hide resolved

adrianreber force-pushed the 2025-07-03-wg-cr branch from 33e97fb to abc1c26 Compare July 9, 2025 12:15

kannon92 reviewed Jul 10, 2025

View reviewed changes

OWNERS_ALIASES Show resolved Hide resolved

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 10, 2025

adrianreber mentioned this pull request Jul 17, 2025

REQUEST: New membership for adrianreber kubernetes/org#5706

Closed

11 tasks

SergeyKanzhelev reviewed Jul 18, 2025

View reviewed changes

sigs.yaml Show resolved Hide resolved

adrianreber force-pushed the 2025-07-03-wg-cr branch from abc1c26 to 8bc6968 Compare July 20, 2025 15:15

k8s-ci-robot added the committee/steering Denotes an issue or PR intended to be handled by the steering committee. label Jul 20, 2025

k8s-ci-robot assigned ritazh Aug 4, 2025

aramase moved this from Needs Triage to In Review in SIG Auth Aug 4, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 5, 2025

viktoriaas mentioned this pull request Aug 11, 2025

REQUEST: New membership for viktoriaas kubernetes/org#5764

Closed

11 tasks

BenTheElder reviewed Aug 20, 2025

View reviewed changes

wg-checkpoint-restore/charter.md Outdated Show resolved Hide resolved

BenTheElder reviewed Aug 20, 2025

View reviewed changes

sigs.yaml Outdated Show resolved Hide resolved

aojea reviewed Aug 21, 2025

View reviewed changes

wg-checkpoint-restore/charter.md Outdated Show resolved Hide resolved

aojea reviewed Aug 21, 2025

View reviewed changes

wg-checkpoint-restore/charter.md Show resolved Hide resolved

viktoriaas reviewed Aug 21, 2025

View reviewed changes

wg-checkpoint-restore/README.md Show resolved Hide resolved

soltysh reviewed Aug 27, 2025

View reviewed changes

adrianreber force-pushed the 2025-07-03-wg-cr branch from 224da25 to 63e2024 Compare September 9, 2025 07:36

k8s-ci-robot removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. labels Sep 9, 2025

adrianreber force-pushed the 2025-07-03-wg-cr branch from 63e2024 to 6df1e19 Compare September 9, 2025 07:41

Introduce WG Checkpoint Restore

da35847

Co-authored-by: Viktória Spišaková <[email protected]> Co-authored-by: Antonio Ojea <[email protected]> Co-authored-by: Sergey Kanzhelev <[email protected]> Signed-off-by: Adrian Reber <[email protected]>

adrianreber force-pushed the 2025-07-03-wg-cr branch from 6df1e19 to da35847 Compare September 9, 2025 11:06

helayoty moved this from Needs Triage to Backlog in SIG Scheduling Sep 9, 2025

helayoty moved this from Backlog to Needs Review in SIG Scheduling Sep 9, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 9, 2025

Introduce WG Checkpoint Restore #8508

Are you sure you want to change the base?

Introduce WG Checkpoint Restore #8508

Conversation

adrianreber commented Jul 3, 2025

Uh oh!

k8s-ci-robot commented Jul 3, 2025

Uh oh!

k8s-ci-robot commented Jul 3, 2025

Uh oh!

k8s-ci-robot commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

kannon92 commented Jul 10, 2025

Uh oh!

kannon92 commented Jul 15, 2025

Uh oh!

adrianreber commented Jul 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rst0git Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikebrow Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aramase commented Aug 4, 2025

Uh oh!

Uh oh!

BenTheElder commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haircommander commented Aug 20, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rst0git Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

rst0git Jul 19, 2025 •

edited

Loading

mikebrow Sep 9, 2025 •

edited

Loading

BenTheElder commented Aug 20, 2025 •

edited

Loading

rst0git Sep 1, 2025 •

edited

Loading