Suggestions: routing incoming jobs to various IAM roles

[chrisforrette]

I wanted to post and see if anyone might have any good ideas/suggestions here...

TL;DR We have self-hosted runners running on AWS EKS and want to be able to use different IAM roles/service accounts based on the jobs being run and there are some limitations we're butting up against—namely IAM role-chaining limits sessions to 1 hour—and I'm looking for workarounds.

Background:

We're running the actions-runner-controller in an AWS EKS cluster and we want to organize our IAM roles and permissions by the type of job being run (e.g. test, e2e, build, deploy, etc.) and environment access they need (staging, production, etc.) with the initial idea to use the aws-actions/configure-aws-credentials in the workflow YML to explicitly assume the appropriate role.

Problem:

Unfortunately I discovered a limitation here... This workflow means we're "role chaining"—in that the IAM role that we've set up with IRSA used by the RunnerDeployment (serviceAccountName) is assuming the roles being used in the aws-actions/configure-aws-credentials step—and role chaining applies a hard limit of 1 hour for sessions. This is fine for most of our jobs, but we eventually want to start running GitHub Actions-based cron jobs, some of which will exceed one hour.

Ideas:

My first thought is that I just need to make RunnerDeployments for every combination of roles and options we offer. We currently have "small", "medium", and "large" deployments for different CPU/memory configurations, so we would have something combining those with all the different serviceAccountName settings so the role could be assumed directly. We would have for example: small + staging deploy role, medium + staging deploy role, large + staging deploy role, small + production deploy role, etc. This would be a lot and require an additional label in runs-on in the workflow to route properly, but it would work and it's what I'm leaning toward.

The only other thoughts I have are theoretical...

I would love if there were a way to set serviceAccountName dynamically based on incoming jobs but implementing a feature and putting YAML configuration around something like this sounds tricky.

I was also reading about using the environment input in GitHub Actions recently and it would be nice (and probably a much easier feature to support) to be able to route to RunnerDeployments based on that setting.

Other than that, I don't have super in-depth knowledge of AWS IAM, and I'm new to Kubernetes/EKS and GitHub Actions, so I'm wondering if there might be some facilities in any of those that I might be missing that could help.

Does anyone have any other ideas or suggestions?

[harshaisgud]

I am using OIDC(https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) from github which uses https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html. This way workers themselves dont need an IAM role.