unable to provision cluster in another aws account

[rohithvarmayadla]

when trying to create a staging cluster it is getting provisioned in same account where mgmt cluster is,
I am looking to have the staging and production clusters in different accounts and mgmt in different account for isolating the environments, however they all are getting provisions din same account,
also I am looking to add Karpenter, could you guide the steps on this.

[iamahgoub]

Hi @rohithvarmayadla -- thanks for reaching out.
You can provision workload clusters in different accounts by following the steps below:

1. Create an additional ProviderConfig that points to the account where you want to deploy the workload cluster. Original ProviderConfig can be found at: https://github.com/aws-samples/eks-multi-cluster-gitops/blob/main/repos/gitops-system/tools/crossplane/crossplane-aws-provider-config/aws-providerconfig.yaml. The new ProviderConfig will be a bit different from the original one; a role in the workload cluster account has to be assumed -- refer to the following sample for guidance: https://github.com/crossplane-contrib/provider-aws/blob/master/AUTHENTICATION.md#using-assumerole.
2. Change the EKS composition to parameterise providerConfigRef.
3. Pass the name of the new ProviderConfig created at step 1 in the claim of the new cluster at: https://github.com/aws-samples/eks-multi-cluster-gitops/blob/main/repos/gitops-system/clusters-config/template/def/eks-cluster.yaml.

Of course, you will have to create an IAM role in the workload cluster account with trust policy that allows assuming it from the IAM role in the management account used for running Crossplane AWS provider; it should have the required IAM permissions for creating EKS clusters, and its dependencies.

With regard to adding Karpenter, you can follow the same pattern used for other tools (e.g. AWS LBC, EBS CSI driver). The steps at a high level are:

1. Creating a folder for tool under https://github.com/aws-samples/eks-multi-cluster-gitops/tree/main/repos/gitops-system/tools/, and configure the HelmRelease and HelmRepository objects.
2. For corresponding IAM requirements, create a folder under https://github.com/aws-samples/eks-multi-cluster-gitops/tree/main/repos/gitops-system/tools-config, and configure the ServiceAccount, Role, Policy, and RolePolicyAttachment.
3. Create two Kustomization for the two new folders at: https://github.com/aws-samples/eks-multi-cluster-gitops/tree/main/repos/gitops-system/clusters/template'; one of the IAM requirements, and the other for the tool itself. Add a dependency for the Kustomization for the tool on the Kustomization for the IAM requirements.

I will add the two scenarios above to the implementation in the coming days. I will update the issue when I am done.

[iamahgoub]

Karpenter is now added to the implementation.

[rohithvarmayadla]

Hi @iamahgoub , Thank you for the response and adding Karpenter to this.
this is beautiful code on Gitops which is almost end to end implementation.
This is going to help many to on board on to aws with very little effort.
as aws moved to EKS API and deprecated aws-auth functionality, updating this to use latest method would be much more beneficial.
that would make this up to date blue print.