Migrating EKS resources to use another lambda

[mukundjalan]

We have an infrastructure created as below and use CDK in python to manage the CDK deployments

• Create EKS cluster using terraform
• In CDK Stack - Importer
  • Import the cluster with a kubectl_layer (KubectlV29Layer)
  • Extract the KubectlProvider from it
  • Export kubectl_provider.service_token as stack output: ProviderFrameworkEventLambdaArn
  • Export kubectl_provider.role_arn as stack output: KubectlProviderRoleArn
  • Export kubectl_provider.handler_role.role_arn as stack output: HandlerRoleArn
• In CDK Stack - Deployer1
  • Import the kubectl_provider with outputs from Importer

        return eks.KubectlProvider.from_kubectl_provider_attributes(
            scope=self,
            id="KubectlProvider",
            function_arn=self.cluster_attributes.function_arn,
            kubectl_role_arn=self.cluster_attributes.kubectl_role_arn,
            handler_role=iam.Role.from_role_arn(
                scope=self,
                id="HandlerRole",
                role_arn=self.cluster_attributes.handler_role_arn
            )
        )
    

  • Import the cluster using above kubectl_provider
  • Use this cluster to create/update eks.KubernetesManifest objects
• Repeat same in stacks Deployer2, Deployer3, ..., DeployerN

This setup has been working fine for us for past couple of months but we need to make certain enhancements to this so as to make it easier to accomplish this process based on the knowledge we gathered in the past year.

So, now, I want to move this setup to a better way of creating the importer stack. As I cannot use the same export value in the new stack, I created the stack as below retaining the old one until migration is complete

• In CDK Stack - ImporterNew
  • Import the cluster with a kubectl_layer (KubectlV29Layer)
  • Extract the KubectlProvider from it
  • Export kubectl_provider.service_token as stack output: ProviderFrameworkEventLambdaArnNew
  • Export kubectl_provider.role_arn as stack output: KubectlProviderRoleArnNew
  • Export kubectl_provider.handler_role.role_arn as stack output: HandlerRoleArnNew

Now I want the DeployerX stacks to make use of ImporterNew's output rather than the old one and later delete the stack Importer altogether. Hence, I tried

• In CDK Stack - DeployerX
  • Import the kubectl_provider with outputs from ImporterNew

        return eks.KubectlProvider.from_kubectl_provider_attributes(
            scope=self,
            id="KubectlProvider",
            function_arn=self.cluster_attributes_new.function_arn,
            kubectl_role_arn=self.cluster_attributes_new.kubectl_role_arn,
            handler_role=iam.Role.from_role_arn(
                scope=self,
                id="HandlerRole",
                role_arn=self.cluster_attributes_new.handler_role_arn
            )
        )
    

  • Import the cluster using above kubectl_provider
  • Use this cluster to create/update eks.KubernetesManifest objects

When synthesising I see output like

 ├─ [~] RoleArn (may cause replacement)
 │   └─ [~] .Fn::ImportValue:
 │       ├─ [-] KubectlProviderRoleArn
 │       └─ [+] KubectlProviderRoleArnNew
 └─ [~] ServiceToken (may cause replacement)
     └─ [~] .Fn::ImportValue:
         ├─ [-] ProviderFrameworkEventLambdaArn
         └─ [+] ProviderFrameworkEventLambdaArnNew
[~] Custom::AWSCDK-EKS-KubernetesResource CoreNamespace/ManifestCoreNamespace/Resource CoreNamespaceManifestCoreNamespaceC0F0AFE9 may be replaced

and while deployment I get error in stack deployment and deployment is rolled back

Deployer1 | 4/6 | 10:53:27 AM | UPDATE_FAILED        | Custom::AWSCDK-EKS-KubernetesResource | CoreNamespace/ManifestCoreNamespace/Resource/Default (CoreNamespaceManifestCoreNamespaceC0F0AFE9) Modifying service token is not allowed.

How can I achieve this migration? It is OK if the manifests are overridden as we have overwrite=True for all manifest objects but deletion and re-creation will not be useful as we have several resources deleting which can make our application unstable.