You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I installed the AWS load balancer controller using Helm and it fails to provision an ALB on AWS when I create new ingresses.
Steps to reproduce
For most of the steps, I was following these guide 1, guide 2, and guide 3.
First of all. I associated the IAM OIDC Provider with my EKS cluster using eksctl utils associate-iam-oidc-provider --region=us-east-1 --cluster=<name> --approve
Roles & Policies
I used this terraform file to create the required roles and policies, and install the helm chart:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
helm = {
source = "hashicorp/helm"
version = "~> 2.9"
}
}
required_version = ">= 1.5.0"
}
provider "aws" {
region = "us-east-1" # Change to your desired region
profile = "default" # Change to your AWS CLI profile if necessary
}
variable "cluster_name" {
description = "The name of the EKS cluster"
type = string
}
variable "vpc_id" {
description = "The ID of the VPC"
type = string
}
data "aws_iam_policy_document" "aws_lbc" {
statement {
effect = "Allow"
principals {
type = "Service"
identifiers = ["pods.eks.amazonaws.com"]
}
actions = [
"sts:AssumeRole",
"sts:TagSession"
]
}
}
resource "aws_iam_role" "aws_lbc" {
name = "AmazonEKSLoadBalancerControllerRole"
assume_role_policy = data.aws_iam_policy_document.aws_lbc.json
}
# I tried this command too: (aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://AWSLoadBalancerControllerIAMPolicy.json)
resource "aws_iam_policy" "aws_lbc" {
# curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/install/iam_policy.json
policy = file("./iam/AWSLoadBalancerControllerIAMPolicy.json")
name = "AWSLoadBalancerControllerIAMPolicy"
}
resource "aws_iam_role_policy_attachment" "aws_lbc" {
policy_arn = aws_iam_policy.aws_lbc.arn
role = aws_iam_role.aws_lbc.name
}
resource "aws_eks_pod_identity_association" "aws_lbc" {
cluster_name = var.cluster_name
namespace = "kube-system"
service_account = "aws-load-balancer-controller"
role_arn = aws_iam_role.aws_lbc.arn
}
data "aws_eks_cluster" "eks" {
name = "<name>"
}
data "aws_eks_cluster_auth" "eks" {
name = "name>"
}
provider "helm" {
kubernetes {
host = data.aws_eks_cluster.eks.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.eks.token
}
}
# I deployed this after creating the service account
resource "helm_release" "aws_lbc" {
name = "aws-load-balancer-controller"
repository = "https://aws.github.io/eks-charts"
chart = "aws-load-balancer-controller"
namespace = "kube-system"
version = "1.11.0"
cleanup_on_fail = true
set {
name = "clusterName"
value = var.cluster_name
}
set{
name = "serviceAccount.create"
value = "false"
}
set {
name = "serviceAccount.name"
value = "aws-load-balancer-controller"
}
set {
name = "vpcId"
value = var.vpc_id
}
set{
name = "region"
value = "us-east-1"
}
set{
name = "replicaCount"
value = 1
}
}
I also tried this AmazonEKSLoadBalancerControllerRole trust relationship:
An ALB should be created and I should see an ALB assigned to my ingress resources.
Current outcome
In my ingress, I got: Failed build model due to operation error Elastic Load Balancing v2: DescribeLoadBalancers, get identity: get credentials: failed to refresh cached credentials, failed to load credentials, exceeded maximum number of attempts, 10, request send failed, Get "http://169.254.170.23/v1/credentials": dial tcp 169.254.170.23:80: i/o timeout
In the controller: {"level":"error","ts":"2025-01-21T20:37:36Z","msg":"Reconciler error","controller":"ingress","object":{"name":"new-shared-k8s-alb-group"},"namespace":"","name":"new-shared-k8s-alb-group","reconcileID":"18b4f2a0-0d46-46b3-a89a-e43cf2c58dd1","error":"operation error Elastic Load Balancing v2: DescribeLoadBalancers, get identity: get credentials: failed to refresh cached credentials, failed to load credentials, exceeded maximum number of attempts, 10, request send failed, Get \"http://169.254.170.2/ and 2025/01/21 20:49:32 http: TLS handshake error from 10.0.21.244:35898: EOF
Then I installed cert-manager using kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.0/cert-manager.yaml and the error became {"level":"error","ts":"2025-01-21T21:48:06Z","msg":"Reconciler error","controller":"ingress","object":{"name":"new-shared-k8s-alb-group"},"namespace":"","name":"new-shared-k8s-alb-group","reconcileID":"6cbe3d55-9104-469f-855f-c1d279b76d36","error":"operation error Elastic Load Balancing v2: DescribeLoadBalancers, get identity: get credentials: failed to refresh cached credentials, failed to load credentials, exceeded maximum number of attempts, 10, request send failed, Get \"http://169.254.170.23/v1/credentials\": dial tcp 169.254.170.23:80: i/o timeout"}
Environment
AWS Load Balancer controller version v2.11.0
Chart version 1.11.0
Using EKS (yes/no), if so version? v1.25.16-eks-2d5f260
Additional Context
I installed these CRDs too:
wget https://raw.githubusercontent.com/aws/eks-charts/master/stable/aws-load-balancer-controller/crds/crds.yaml
kubectl apply -f crds.yaml
# I tried this too
kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master"
The text was updated successfully, but these errors were encountered:
I had the same issue, appears that in my ingress definition I had to use the old annotation and not the new one: kubernetes.io/ingress.class: alb - works (despite de deprecation warning when applied) kubernetes.io/ingressClassName: alb - does not create any loadbalancer ! Spend a few hours debugging and verifying everything before I tried this older annotation. Please fix
I installed the AWS load balancer controller using Helm and it fails to provision an ALB on AWS when I create new ingresses.
Steps to reproduce
For most of the steps, I was following these guide 1, guide 2, and guide 3.
First of all. I associated the IAM OIDC Provider with my EKS cluster using
eksctl utils associate-iam-oidc-provider --region=us-east-1 --cluster=<name> --approve
Roles & Policies
I used this terraform file to create the required roles and policies, and install the helm chart:
I also tried this
AmazonEKSLoadBalancerControllerRole
trust relationship:Furthermore, I tried installing the helm chart using the helm cli instead of terraform:
Ingress
This is my ingress manifest:
Service Account
I created a service account before installing the chart using:
It created the service account with this auto-generated annotation
It didn't work, so I updated it to
eks.amazonaws.com/role-arn
toarn:aws:iam::<>:role/AmazonEKSLoadBalancerControllerRole
:unfortunately, it didn't make any difference.
Expected outcome
An ALB should be created and I should see an ALB assigned to my ingress resources.
Current outcome
Failed build model due to operation error Elastic Load Balancing v2: DescribeLoadBalancers, get identity: get credentials: failed to refresh cached credentials, failed to load credentials, exceeded maximum number of attempts, 10, request send failed, Get "http://169.254.170.23/v1/credentials": dial tcp 169.254.170.23:80: i/o timeout
{"level":"error","ts":"2025-01-21T20:37:36Z","msg":"Reconciler error","controller":"ingress","object":{"name":"new-shared-k8s-alb-group"},"namespace":"","name":"new-shared-k8s-alb-group","reconcileID":"18b4f2a0-0d46-46b3-a89a-e43cf2c58dd1","error":"operation error Elastic Load Balancing v2: DescribeLoadBalancers, get identity: get credentials: failed to refresh cached credentials, failed to load credentials, exceeded maximum number of attempts, 10, request send failed, Get \"http://169.254.170.2/
and2025/01/21 20:49:32 http: TLS handshake error from 10.0.21.244:35898: EOF
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.0/cert-manager.yaml
and the error became{"level":"error","ts":"2025-01-21T21:48:06Z","msg":"Reconciler error","controller":"ingress","object":{"name":"new-shared-k8s-alb-group"},"namespace":"","name":"new-shared-k8s-alb-group","reconcileID":"6cbe3d55-9104-469f-855f-c1d279b76d36","error":"operation error Elastic Load Balancing v2: DescribeLoadBalancers, get identity: get credentials: failed to refresh cached credentials, failed to load credentials, exceeded maximum number of attempts, 10, request send failed, Get \"http://169.254.170.23/v1/credentials\": dial tcp 169.254.170.23:80: i/o timeout"}
Environment
Additional Context
I installed these CRDs too:
The text was updated successfully, but these errors were encountered: