✨ IPv6 support for self-managed clusters #5603

tthvo · 2025-07-30T21:03:14Z

What type of PR is this?
/kind feature

What this PR does / why we need it:

As of today, CAPA supports IPv6 on EKS, but not self-managed clusters. Thus, these changes bring IPv6 support for self-managed clusters, both single-stack IPv6 and dualstack.

Which issue(s) this PR fixes:

Fixes #2420
Fixes #3381

Special notes for your reviewer:

EC2 instance type: Only nitro-based instance type can support IPv6.
CNI: The CNI plugin needs to support IPV6. I include sample manifests in test/e2e/data/cni. Calico does not support IPv6 with "IP-in-IP" so we need to use VXLAN.

Checklist:

Release note:

Enable IPv6 support for self-managed kubernetes clusters

linux-foundation-easycla · 2025-07-30T21:03:20Z

The committers listed above are authorized under a signed CLA.

✅ login: tthvo / name: Thuan Vo (04d4286, 061134a, 1ebffda, 2c7e1e4, 2fb4f55, 3292f75, 4d38ed3, 4e7e9a7, 60308b2, 7826e7e, 78cb9d4, 7949ccb, 7bbae60, 842f85a, 9851222, 987bc9c, b8ca23a, bcda5bf, bfbbb84, c1bafb0, cd6c178, cd7c9b2, da96854, def75f2, e5e3ad3, ea41d67)

k8s-ci-robot · 2025-07-30T21:03:21Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nrb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-07-30T21:03:23Z

Welcome @tthvo!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-07-30T21:03:24Z

Hi @tthvo. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tthvo · 2025-07-30T21:23:21Z

/cc @nrb @sadasu @patrickdillon

I am not yet sure what to do with e2e tests or if there are any existing ones for IPv6 clusters...I leave it as an pending TODO.

k8s-ci-robot · 2025-07-30T21:23:24Z

@tthvo: GitHub didn't allow me to request PR reviews from the following users: sadasu, patrickdillon.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @nrb @sadasu @patrickdillon

I am not yet sure what to do with e2e tests or if there are any existing ones for IPv6 clusters...I leave it as an pending TODO.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tthvo · 2025-07-30T21:33:01Z

A quick preview of kubectl get for an IPv6 self-managed cluster can be found here. So far, nodes, pods and services are assigned the expected IPv6 family IPs 😄

damdo · 2025-08-04T09:45:03Z

/ok-to-test

damdo · 2025-08-04T09:46:01Z

/assign @mtulio

Asking you for a review Marco as I know you have been working on this downstream

api/v1beta2/defaults.go

docs/book/src/topics/ipv6-enabled-cluster.md

We need an option to configure IPv6 source CIDRs for SSH ingress rule of the bastion host. This extends the field allowedCIDRBlocks to also accepts IPv6 CIDR blocks.

When creating a bastion host for an IPv6 cluster, the instance has both public IPv4 and IPv6. Thus, we need to report them in the cluster status if any. This also adds an additional print column to display that bastion IPv6.

This is a minimal template set to install an IPv6-enabled cluster. Both the controlplane and worker nodes must use nitro-based instance type (with IPv6 support).

This is a set of customized calico CNI manifests to support single-stack IPv6 cluster. Note that VXLAN is used since IP-in-IP currently only supports IPv4. References: - https://docs.tigera.io/calico/latest/networking/ipam/ipv6#ipv6 - https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/config-options#switching-from-ip-in-ip-to-vxlan - https://docs.tigera.io/calico/latest/networking/configuring/vxlan-ipip

This combines existing docs for IPv6 EKS clusters with non-EKS ones, and also properly register the topic page into the documentation TOC.

Validation for specified VPC and subnet CIDRs is added for early feedback from the webhook. There are already existing checks for bastion and nodePort CIDRs.

The following is added: - [BYO VPC] Mention the required route when enabling DNS64. - [BYO VPC] Mention that CAPA only utilizes the IPv6 aspect of the dual stack VPC.

There is a brief period where the IPv6 CIDR is not yet associated with the subnets. Thus, when CAPA creates the default dualstack subnets, it should wait until the IPv6 CIDR is associated before proceeding. If not, CAPA will misinterprete the subnet as non-IPv6 and proceed its reconcilation. The consequence is that CAPA will skip creating a route to eigw. Route to eigw for destination "::/0" to eigw is required for EC2 instance time sync on start-up.

…ined When AWSCluster.spec.network.vpc.ipv6 is non-nil, most handlers in CAPA treats it as "adding" IPv6 capabilities on top of IPv4 infrastructure. Except security group ingress rules for API LB. This commit aligns the API LB SG handler with the rest of the code base. These rules can be overriden in the AWSCluster LB spec to allow only IPv6 CIDRs if needed.

The field isIpv6 is set to true if and only if the subnet has an associated IPv6 CIDR. This means the VPC is also associated with an IPv6 CIDR.

The field targetGroupIPType is added to the loadbalancer spec to allow configuring ip address type of target group for API load balancers. This field is not applicable to Classic Load Balancers (CLB). This commit also defines a new network status field to determine the ip type of API load balancers.

When creating subnets in a managed VPC with IPv6 enabled, automatically assign IPv6 CIDR blocks to subnets that have isIPv6=true but no explicit IPv6CidrBlock specified. This simplifies subnet configuration by allowing users to enable IPv6 without manually calculating and assigning individual subnet IPv6 CIDRs, for example, in case where VPC IPv6 CIDR is unknown pre-provisioning and AWS will assign one during VPC creation. Note: This logic only applies when spec.network.vpc.ipv6 is non-nil, subnets are managed and non-existing.

The field awscluster.spec.network.vpc.ipv6.ipamPool defines the IPAM pool to allocate an IPv6 CIDR for the VPC. Previously, CAPA only considers field awscluster.spec.network.vpc.ipamPool, which is used only for VPC IPv4 CIDR allocation. Additionally, CAPA should preserve the ipv6 spec fields, provided by the users, for example, the ipv6 ipamPool. Previously, these spec fields are lost during vpc reconcilation.

NAT64/DNS64 is meant to be enabled for IPv6-only subnets, in which instances do not have IPv4 [0]. If we enable DNS64 for dualstack subnets, instances will receive both A/AAAA records for IPv4-only services. In most cases, OS-level settings will prefer IPv6, leading to traffic to flow via NAT gateway instead of using IPv4 directly. Reference [0] https://aws.amazon.com/blogs/networking-and-content-delivery/dual-stack-architectures-for-aws-and-hybrid-networks-part-2/

Add new dualstack cluster template and documentation updates for IPv6 and dualstack cluster configurations. Additionally, docs for configuring API LB's target group IP type is also added. New cluster templates and calico manifest are included for creating dualstack clusters.

tthvo · 2025-10-20T23:19:00Z

Date: Oct 20, 2025

Status: The users can now use CAPA to provision IPv6-enabled VPCs (i.e. any mixing of dualstack, Ipv6-only, IPv4-only subnets that meets their need) and install an IPv6/dualstack k8s cluster on top of that network infrastructure (See doc ipv6-enabled-cluster.md).

Limitations: No support for CAPA-provisioned IPv6 edge subnets: wavelength zones do not support IPv6 and local zones require a secondary IPv6 VPC CIDR. See more details in the updated docs above.

I think this PR is ready for another round of reviews 🙏 🙇

/cc @sadasu @mtulio @patrickdillon @damdo @nrb

k8s-ci-robot · 2025-10-20T23:19:07Z

@tthvo: GitHub didn't allow me to request PR reviews from the following users: patrickdillon.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Date: Oct 20, 2025

Status: The users can now use CAPA to provision IPv6-enabled VPCs (i.e. any mixing of dualstack, Ipv6-only, IPv4-only subnets that meets their need) and install an IPv6/dualstack k8s cluster on top of that network infrastructure (See doc ipv6-enabled-cluster.md).

Limitations: No support for CAPA-provisioned IPv6 edge subnets: wavelength zones do not support IPv6 and local zones require a secondary IPv6 VPC CIDR. See more details in the updated docs above.

I think this PR is ready for another round of reviews 🙏 🙇

/cc @sadasu @mtulio @patrickdillon @damdo @nrb

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

pkg/cloud/services/ec2/instances.go

sadasu · 2025-10-22T17:16:50Z

api/v1beta2/types.go

+	//
+	// +kubebuilder:validation:Enum:=enabled;disabled
+	// +kubebuilder:default=disabled
+	HTTPProtocolIPv6 InstanceMetadataState `json:"httpProtocolIpv6,omitempty"`


Nit: Should this be HTTPIPv6Endpoint or HTTPEndpointIPv6 to be consistent with HTTPEndpoint for IPv4?

Ahh, the name HTTPProtocolIPv6 is actually used to match the corresponding AWS metadata option field, for example:

$ aws ec2 modify-instance-metadata-options --instance-id i-1234567890 \ --http-endpoint=disabled --http-protocol-ipv6=enabled { "InstanceId": "i-1234567890", "InstanceMetadataOptions": { "State": "pending", "HttpTokens": "required", "HttpPutResponseHopLimit": 2, "HttpEndpoint": "disabled", "HttpProtocolIpv6": "enabled", "InstanceMetadataTags": "disabled" } }

https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-metadata-options.html

I think that's the convention here in capa 👀 so we should keep it this way?

sadasu · 2025-10-22T17:41:49Z

pkg/cloud/services/network/routetables.go


 	routes = append(routes, s.getNatGatewayPrivateRoute(natGatewayID))
 	if sn.IsIPv6 {
+		routes = append(routes, s.getNat64PrivateRoute(natGatewayID))


Not sure why this was not needed for managed installs?

Do we need to check for !sn.IsPublic here?

Do we need to check for !sn.IsPublic here?

I think the check is already done in

cluster-api-provider-aws/pkg/cloud/services/network/routetables.go

Lines 429 to 434 in 6f867be

func (s *Service) getRoutesForSubnet(sn *infrav1.SubnetSpec) ([]*ec2.CreateRouteInput, error) {

if sn.IsPublic {

return s.getRoutesToPublicSubnet(sn)

}

return s.getRoutesToPrivateSubnet(sn)

}

The func getRoutesToPrivateSubnet is only ever used for getting routes for private subnets :D

Not sure why this was not needed for managed installs

This route is actually a component of DNS64 (NAT64 is always on) to resolve an IPv4-only internet service to a synthetic IPv6 (See here). It is really a nice-to-have feature 🤔 but not required.

My guess was that when Ipv6 was supported for managed eks, they didn't need IPv6-only pods to access IPv4-only internet services.

Additionally, I think CAPA currently support IPv6 for EKS by using dualstack subnets. And NAT64/DNS64 is really meant for IPv6-only subnets, not dualstack subnets (unless really necessary).

See details in commit: 78cb9d4

nrb · 2025-10-23T16:01:27Z

api/v1beta2/awscluster_webhook.go

+		}
+	}
+
+	if r.Spec.ControlPlaneLoadBalancer != nil {


Do we need this check here? L435 does it as well. Should the conditionals within this block move into that one?

Oh sure, I just wanted to separate this check into a block for easier read. I can refactor a bit more.

yeah, the block above is pretty big. Maybe we could invert the logic to skip things if the value's nil, skip all the other checks? Might not make sense in context, unsure.

Than ks, I did a bit of refactoring to unify the nil check 👀

Since we need to check both primary and secondary LB settings, we probably should still stick with != nil. If non-nil, check and skip otherwise - seems a bit cleaner...

docs/book/src/topics/ipv6-enabled-cluster.md

nrb · 2025-10-23T16:14:27Z

docs/book/src/topics/network-load-balancer-with-awscluster.md

+
+### Additional Listeners
+
+The `targetGroupIPType` can also be configured independently for each additional listener:


Do you have an example use case for this? Is it to keep external (to the cluster) traffic IPv4 while allowing internal traffic to be IPv6?

Oh yes, that is one of the potential use case for single-stack IPv6 (where AWS NLBs only support dualstack). Other cases I have in mind:

For dualstack where IPv4 is the primary family, the users (case 2 below) might want to configure the target group as IPv4.

As an IPv6 target group requires EC2 instances to have a "primary IPv6", which must be enabled explicitly. For OpenShift, MAPI currently does not support "primary IPv6". So, we'd want to use case 1.

docs/book/src/topics/secondary-load-balancer.md

pkg/cloud/services/ec2/instances.go

pkg/cloud/services/network/subnets.go

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. labels Jul 30, 2025

k8s-ci-robot requested review from AndiDog and damdo July 30, 2025 21:03

k8s-ci-robot added needs-priority cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 30, 2025

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 30, 2025

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 30, 2025

k8s-ci-robot requested a review from nrb July 30, 2025 21:23

tthvo force-pushed the singestack-ipv6 branch from ddacd99 to 13e4379 Compare July 31, 2025 00:37

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 4, 2025

k8s-ci-robot assigned mtulio Aug 4, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 4, 2025

tthvo force-pushed the singestack-ipv6 branch from 13e4379 to 8816d87 Compare August 5, 2025 18:09

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 5, 2025

tthvo force-pushed the singestack-ipv6 branch from 8816d87 to 5dd2888 Compare August 5, 2025 18:11

nrb reviewed Aug 5, 2025

View reviewed changes

api/v1beta2/defaults.go Outdated Show resolved Hide resolved

nrb reviewed Aug 5, 2025

View reviewed changes

docs/book/src/topics/ipv6-enabled-cluster.md Outdated Show resolved Hide resolved

tthvo added 16 commits October 20, 2025 14:02

securitygroup: allow configuring IPv6 source CIDRs for bastion SSH

cd6c178

We need an option to configure IPv6 source CIDRs for SSH ingress rule of the bastion host. This extends the field allowedCIDRBlocks to also accepts IPv6 CIDR blocks.

crd: add IPv6 of bastion host to cluster status

1ebffda

When creating a bastion host for an IPv6 cluster, the instance has both public IPv4 and IPv6. Thus, we need to report them in the cluster status if any. This also adds an additional print column to display that bastion IPv6.

template: manifest templates for IPv6-enabled cluster

2c7e1e4

This is a minimal template set to install an IPv6-enabled cluster. Both the controlplane and worker nodes must use nitro-based instance type (with IPv6 support).

docs: add documentations for enabling IPv6 in non-eks clusters

cd7c9b2

This combines existing docs for IPv6 EKS clusters with non-EKS ones, and also properly register the topic page into the documentation TOC.

validations: validate vpc and subnet CIDR

def75f2

Validation for specified VPC and subnet CIDRs is added for early feedback from the webhook. There are already existing checks for bastion and nodePort CIDRs.

docs: update doc for enabling ipv6

9851222

The following is added: - [BYO VPC] Mention the required route when enabling DNS64. - [BYO VPC] Mention that CAPA only utilizes the IPv6 aspect of the dual stack VPC.

cni: document the requirement for calico ipv6 support

ea41d67

crd: clarify isIpv6 field on subnet spec

e5e3ad3

The field isIpv6 is set to true if and only if the subnet has an associated IPv6 CIDR. This means the VPC is also associated with an IPv6 CIDR.

tthvo force-pushed the singestack-ipv6 branch from 9edbe77 to 061134a Compare October 20, 2025 23:03

k8s-ci-robot requested review from mtulio, nrb and sadasu October 20, 2025 23:19

sadasu reviewed Oct 22, 2025

View reviewed changes

pkg/cloud/services/ec2/instances.go Show resolved Hide resolved

sadasu reviewed Oct 22, 2025

View reviewed changes

nrb reviewed Oct 23, 2025

View reviewed changes

tthvo added 3 commits October 23, 2025 14:23

fixup! ec2: enable primary IPv6 on ENI for EC2 instances

bcda5bf

fixup! docs: add dualstack cluster support documentation

4e7e9a7

fixup! api: add spec field to configure target group ipType

2fb4f55

tthvo force-pushed the singestack-ipv6 branch from 4701b61 to 2fb4f55 Compare October 23, 2025 21:39

	func (s Service) getRoutesForSubnet(sn infrav1.SubnetSpec) ([]*ec2.CreateRouteInput, error) {
	if sn.IsPublic {
	return s.getRoutesToPublicSubnet(sn)
	}
	return s.getRoutesToPrivateSubnet(sn)
	}


		### Additional Listeners

		The `targetGroupIPType` can also be configured independently for each additional listener:

✨ IPv6 support for self-managed clusters #5603

Are you sure you want to change the base?

✨ IPv6 support for self-managed clusters #5603

Conversation

tthvo commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linux-foundation-easycla bot commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

tthvo commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

tthvo commented Jul 30, 2025

Uh oh!

damdo commented Aug 4, 2025

Uh oh!

damdo commented Aug 4, 2025

Uh oh!

Uh oh!

Uh oh!

tthvo commented Oct 20, 2025

Uh oh!

k8s-ci-robot commented Oct 20, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

tthvo commented Jul 30, 2025 •

edited

Loading

linux-foundation-easycla bot commented Jul 30, 2025 •

edited

Loading