Allow K8s Service to override controller default tags by annotation

[jtackaberry]

Is your feature request related to a problem?

We define default tags at the AWS LB controller level, with the intention that these tags should be used as, well, defaults for the environment if the LoadBalancer type K8s Service doesn't specify service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags.

Unfortunately that annotation doesn't override default tags, which I understand is clearly documented but doesn't actually provide the behavior we need, which is that default tags are defaults unless they are overridden by more granular resources.

Describe the solution you'd like

Even though IMO this violates the Principle of Least Astonishment, I accept that the behavior of service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags is by now well established and probably can't change without disruption, so I propose a new Service annotation service.beta.kubernetes.io/aws-load-balancer-resource-tags-overrides or some such, where the defaults defined at the controller level are used but can be overridden by any tags specified in this new annotation.

Describe alternatives you've considered

We'll have to drop default tags entirely from the AWS LB controller and instead vigilantly police all K8s Service objects for compliance instead. This isn't ideal.

[andreybutenko]

Hi! Thank you for the issue

service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags is an annotation that enables additional tags. Today, there is no annotation to override tags at the service level

Could you tell us more about your usecase, why you'd like to override tags at the service level? Thanks!

[jtackaberry]

Hi @andreybutenko

service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags is an annotation that enables additional tags. Today, there is no annotation to override tags at the service level

Yep, understood, which is why I suggested a new annotation, one which can introduce new semantics without affecting use of the existing annotation.

Could you tell us more about your usecase, why you'd like to override tags at the service level?

Sure.

At my organization, we often have EKS clusters which host multiple services that have different billing attributes and/or different support teams. It's an internal requirement that all cloud resources contain specific tags to indicate contact information and service-level information (an identifier we call a service record). The more granular the tag values the better -- contact details for specific teams rather than departmental (or platform team) contacts, specific services rather than higher level service families.

But should the team managing deployments for a specific service neglect to add the required annotation, we need to fall back to the less granular tag values. For example, if we have a security incident, the SOC must have a clear path of escalation. Preferably, the service's deployment includes those details, but if not, the platform team that manages the broader infrastructure is the next line of contact.

So the basic idea is that the LB controller would define as default tags those higher level values for contact and service family, something appropriate for the larger scope of the EKS cluster into which the AWS LB Controller is deployed. Then, if a LoadBalancer type Service is created that lacks more granular overrides, at least something will be populated to provide a reasonable starting point for escalations and billing aggregations. In certain cases, the LoadBalancer actually is a general shared LB such that those high level tags make sense, so we would just leave the defaults, but in other cases the defaults are better overridden.

Hope that makes sense. Let me know if I can help to clarify anything.