[Feature Request] PrivateLink Support

[rifelpet]

We have a need to create PrivateLink Endpoint Services with our NLBs. A Service resource would correspond to one NLB as well as one Endpoint Service configured to use that NLB.

I imagine the annotation design would be something like:

service.beta.kubernetes.io/aws-load-balancer-endpoint-service-acceptance-required: 'true'
service.beta.kubernetes.io/aws-load-balancer-endpoint-service-allowed-principals: '["arn:aws:iam::123456789012:root"]'
service.beta.kubernetes.io/aws-load-balancer-endpoint-service-private-dns-name: 'foobar.example.com'

I don't have a good idea for how to conditionally add the DNS verification record... typically external-dns would handle that but given that both the name and value are computed, it doesn't align as well with the external-dns pattern of reading the name from a spec or annotation and the value from a resource's status or annotation.

A stretch goal would be to support creating Endpoints in a specified set of VPCs but given that (in our case) many of the VPCs would be in different AWS accounts and aws-load-balancer-controller doesn't support using multiple credentials for multiple accounts, I understand if we don't want to incorporate that complexity directly into aws-load-balancer-controller.

Would a PR for supporting Endpoint Services be accepted in the load balancer controller? Or are there any existing plans to add something similar?

[fandujar]

@rifelpet I need something similar. I got some issues when deleting a load balancer associated with a VPC Endpoint Service.

Maybe it would worth adding an annotation to your request to reuse a VPC Endpoint Service:
service.beta.kubernetes.io/aws-load-balancer-endpoint-service-id: 'vpce-svc-xxxxxx'

That way you can manage your VPC Endpoint Service without add complexity to aws-load-balancer-controller and still manage this association lifecycle of a load balancer with private link.

[M00nF1sh]

@fandujar @rifelpet

We don't have plans internally for this. PRs are always welcome, however given this is a little complicated, i think we need a placeholder PR finalize the interfaces first before any code implementations, which we can review internally along with community review.

Personally i'd favor the aws-load-balancer-endpoint-service-id options if we can create a endpointService without NLB. (however seems an NLB is required when create a endpointService. I'm not familiar with privateLink so i might be wrong).

For the DNS validation, i don't think we can do much from the controller as it don't have control over the DomainNames. (we could modify it if it's owned by route53 like coreDNS, but seems it's over complicated for this controller's purpose). how about expose the validation records via annotation?

[rifelpet]

You are correct that ec2.CreateVpcEndpointServiceConfiguration requires that either an NLB or GWLB is provided, though you can later disassociate them from the endpoint service, resulting in an endpoint service with no load balancers behind it.

I think an endpoint-service-id annotation would only be useful for the following workflows:

1. Create Service resource in k8s (load-balancer-controller creates NLB)
2. Create VPC Endpoint Service externally, providing the new NLB's ARN
3. Add endpoint-service-id annotation to existing Service resource
4. When Service resource is deleted, aws-load-balancer-controller first detaches it from the endpoint service

or

1. Create VPC Endpoint Service externally attaching to some other NLB
2. Create Service resource in k8s with endpoint-service-id annotation for the new NLB to be attached to the existing endpoint service
3. When Service resource is deleted, aws-load-balancer-controller first detaches the NLB from the endpoint service

The first use case would likely be made easier if the aws-load-balancer-controller could manage the entire lifecycle of the endpoint service, which is what my original feature request addresses. The second use-case would be useful for endpoint services with many NLBs attached. If aws-load-balancer-controller were to manage the endpoint services for this use-case it could likely use an equivalent of the IngressGroup CRD but for NLBs behind an endpoint service (EndpointServiceGroup?), I'm not planning on adding support for this.

I'm going to plan on moving forward with my proposal for having aws-load-balancer-controller create and manage the endpoint service resources, though much of the work could be reused in supporting an endpoint-service-id annotation. I'll have an interface PR draft open soon for review. I agree that aws-load-balancer-controller shouldn't handle the DNS validation but could expose that information via annotations.

[M00nF1sh]

just thought about this again, i agree with you. personally i think we can expose the created service-id via annotation instead of validation informations. users can have automation tools to read the validation from the service-id

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[rifelpet]

/remove-lifecycle stale

@M00nF1sh any update on this or reviewing the interface proposed in the linked PR?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[rifelpet]

/remove-lifecycle stale

[Cyanopus]

This is a very important business case. Adding a private link as a single annotation is a great way to expose your service B2B without network-level trust.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[hintofbasil]

/remove-lifecycle stale

[nijave]

It'd also be useful if removing load balancers/deleting PrivateLinks on delete is possible. Right now, if you add a PrivateLink and try to delete the load balancer in Kubernetes, it gets stuck deleting the load balancer due to the associated PrivateLink

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Cyanopus]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[valeriodelsarto]

Is the PR #2636 that aim to add this feature still WIP? Seems stale since a few months... having this feature will help a lot managing PrivateLink Endpoint Services from NLBs using only K8s resources

[eminugurkenar]

yes , having this is very useful for us too, would be good to have it on next releases.

[gosunilgo]

bumping this request - What tasks are left to release-it / get-over-hump? reminder pr #2636 from @valeriodelsarto

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kjenney]

/remove-lifecycle stale

[kjenney]

This issue is blocking me at the moment. I see that the last comment in the PR was 2 days ago. Is there any way to get this PR merged ASAP?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kjenney]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kjenney]

/remove-lifecycle stale

[m17kea]

Any update on this?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[m17kea]

/remove-lifecycle stale

[rameshavutu]

/remove-lifecycle stale

[serhatcetinkaya]

any updates on this?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[m17kea]

/remove-lifecycle stale