Runner not installing due to webhook error

[leovanhaaren]

Hey,

I'm facing a issue which I've seen before;

Error from server (InternalError): error when creating "runner.yaml": Internal error occurred: failed calling webhook "mutate.runnerdeployment.actions.summerwind.dev": Post "https://actions-runner-controller-webhook.actions-runner-system.svc:443/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment?timeout=30s": service "actions-runner-controller-webhook" not found

controller-manager logs:

2021-02-19T10:56:29.669Z ERROR controller-runtime.controller Reconciler error {"controller": "runner", "request": "actions-runner-system/jellow-runner-dqgq8-qb2lw", "error": "Internal error occurred: failed calling webhook \"mutate.runner.actions.summerwind.dev\": Post \"https://actions-runner-controller-webhook.actions-runner-system.svc:443/mutate-actions-summerwind-dev-v1alpha1-runner?timeout=30s\": service \"actions-runner-controller-webhook\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
 /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
 /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
 /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
 /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88

I've already done the following, which helped before but it seems it doesn't now:

kubectl delete validatingwebhookconfiguration validating-webhook-configuration                     
validatingwebhookconfiguration.admissionregistration.k8s.io "validating-webhook-configuration" deleted

kubectl delete mutatingwebhookconfiguration mutating-webhook-configuration                         
mutatingwebhookconfiguration.admissionregistration.k8s.io "mutating-webhook-configuration" deleted

The weird thing is that this approach worked before, and I've followed the advice here:
#249 (comment)

Also installing the controller-manager without the webhooks does work.

I'm running the cluster over at DigitalOcean, Scaleway does not seem to have the issue. Any ideas?

[leovanhaaren]

Hmm interesting, installing the helm chart seems to work, but the yaml from v0.17 release does not.

https://github.com/summerwind/actions-runner-controller/releases/download/v0.17.0/actions-runner-controller.yaml

[mumoshu]

@leovanhaaren Yeah interesting. The error still says you have webhook configuration(s) existing on the cluster. To double-check, (and if you have not done yet), could you run kubectl get mutatingwebhookconfiguration and see if there're remaining configs?

[andoriyu]

Hmm interesting, installing the helm chart seems to work, but the yaml from v0.17 release does not.

https://github.com/summerwind/actions-runner-controller/releases/download/v0.17.0/actions-runner-controller.yaml

Installing from helm chart gave me the same error by the way.

[mumoshu]

@andoriyu Hey! Could you provide me the exact steps to reproduce it and the exact error message you've seen?

[andoriyu]

@andoriyu Hey! Could you provide me the exact steps to reproduce it and the exact error message you've seen?

Sure thing.

• Install (via helm3) chart 0.5.2 (values at the bottom)
• Apply runner (manifest at the bottom)
• Wait
• Observe an error:

Error from server (InternalError): error when creating "example-org-runner.yaml": Internal error occurred: failed calling webhook "mutate.runner.actions.summerwind.dev": Post https://general-1-gh-runners-actions-runner-controller-webhook.gh-runners.svc:443/mutate-actions-summerwind-dev-v1alpha1-runner?timeout=30s: context deadline exceeded 

If I run k delete ValidatingWebhookConfiguration general-1-gh-runners-actions-runner-controller-validating-webhook-configuration during apply stage it creates the runner as expected and is working.

helm values:

actions-runner-controller:
  authSecret:
    enabled: false # created manually
  image:
    tag: v0.17.0 # lock the latest version
  githubWebhookServer: 
    enabled: false

Runner manifest:

apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: example-org-runner
spec:
  organization: my-org

[mumoshu]

@andoriyu Thanks!

Apparently, it is different than the original issue. The original issue was service "actions-runner-controller-webhook" not found which usually shouldn't happen/can occur only when you somehow missed installing the k8s service resource at all.

Yours happens when e.g.:

• Some corporate proxy is blocking your controller-manager from working so there's no K8s endpoint backing the K8s service general-1-gh-runners-actions-runner-controller-webhook
• controller-manager is failing to start at pull due to invalid tag/repo/docker registry mirror setting
• Anything that prevents a standard K8s service from working

So, would you firstly share me the result of kubectl describe po $POD and kubectl describe svc $SVC, where $POD is the pod name created for your controller-manager, and $SVC is the name of the K8s service that isn't working.

[andoriyu]

@andoriyu Thanks!

Apparently, it is different than the original issue. The original issue was service "actions-runner-controller-webhook" not found which usually shouldn't happen/can occur only when you somehow missed installing the k8s service resource at all.

Yours happens when e.g.:

* Some corporate proxy is blocking your controller-manager from working so there's no K8s endpoint backing the K8s service `general-1-gh-runners-actions-runner-controller-webhook`

* controller-manager is failing to start at pull due to invalid tag/repo/docker registry mirror setting

* Anything that prevents a standard K8s service from working

So, would you firstly share me the result of kubectl describe po $POD and kubectl describe svc $SVC, where $POD is the pod name created for your controller-manager, and $SVC is the name of the K8s service that isn't working.

Oh, my bad. When I originally investigated this - I was able to curl webhook service btw.

Here is service:

Name:              general-1-gh-runners-actions-runner-controller-webhook
Namespace:         gh-runners                                                                                                                         Labels:            app.kubernetes.io/instance=general-1-gh-runners                                                                                                       app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=actions-runner-controller
                   helm.sh/chart=actions-runner-controller-0.5.2
Annotations:       <none>
Selector:          app.kubernetes.io/instance=general-1-gh-runners,app.kubernetes.io/name=actions-runner-controller
Type:              ClusterIP
IP:                10.120.0.219
Port:              https  443/TCP                                                                                                                     TargetPort:        9443/TCP
Endpoints:         10.100.0.40:9443
Session Affinity:  None
Events:            <none>

Name:         general-1-gh-runners-actions-runner-controller-7846f9c9f8-rz8tk
Namespace:    gh-runners
Priority:     0
Node:         gke-general-1-default-43a878e3-b17m/10.60.0.60
Start Time:   Sat, 20 Feb 2021 01:00:23 +0000
Labels:       app.kubernetes.io/instance=general-1-gh-runners
              app.kubernetes.io/name=actions-runner-controller
              pod-template-hash=7846f9c9f8
Annotations:  cni.projectcalico.org/podIP: 10.100.0.40/32
Status:       Running
IP:           10.100.0.40
IPs:
  IP:           10.100.0.40
Controlled By:  ReplicaSet/general-1-gh-runners-actions-runner-controller-7846f9c9f8
Containers:
  manager:
    Container ID:  docker://738a02e91027df471f6df124e2c744406a599a458aa201e039aed9bf60051cb2
    Image:         summerwind/actions-runner-controller:v0.17.0
    Image ID:      docker-pullable://summerwind/actions-runner-controller@sha256:a1509cb9247a52b37709a7e1c9c6b800c90eb33a07b415e67b5c4c6e5d6e4e83
    Port:          9443/TCP
    Host Port:     0/TCP
    Command:
      /manager
    Args:
      --metrics-addr=127.0.0.1:8080
      --enable-leader-election
      --sync-period=10m
      --docker-image=docker:dind
    State:          Running
      Started:      Sat, 20 Feb 2021 01:00:24 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      GITHUB_TOKEN:                <set to the key 'github_token' in secret 'controller-manager'>                Optional: true
      GITHUB_APP_ID:               <set to the key 'github_app_id' in secret 'controller-manager'>               Optional: true
      GITHUB_APP_INSTALLATION_ID:  <set to the key 'github_app_installation_id' in secret 'controller-manager'>  Optional: true
      GITHUB_APP_PRIVATE_KEY:      /etc/actions-runner-controller/github_app_private_key
    Mounts:
      /etc/actions-runner-controller from controller-manager (ro)
      /tmp from tmp (rw)
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from general-1-gh-runners-actions-runner-controller-token-rnzs5 (ro)
  kube-rbac-proxy:
    Container ID:  docker://c36c911fada14a2067e223968b4f96ec9ea69b43b14f499c3fec15904f95a7ba
    Image:         quay.io/brancz/kube-rbac-proxy:v0.8.0
    Image ID:      docker-pullable://quay.io/brancz/kube-rbac-proxy@sha256:05e15e1164fd7ac85f5702b3f87ef548f4e00de3a79e6c4a6a34c92035497a9a
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=10
    State:          Running
      Started:      Sat, 20 Feb 2021 01:00:24 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from general-1-gh-runners-actions-runner-controller-token-rnzs5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  controller-manager:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  controller-manager
    Optional:    false
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  webhook-server-cert
    Optional:    false
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  general-1-gh-runners-actions-runner-controller-token-rnzs5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  general-1-gh-runners-actions-runner-controller-token-rnzs5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

[andoriyu]

@mumoshu so any ideas what's the issue?

[mumoshu]

@andoriyu Unfortunately no. Your information indicates that everything's working well...

To be extra sure, have you tried holding on for a few seconds to a minute before applying example-org-runner.yaml? Depending on the environment, it can take some time until the admission webhook gets fully set up. And trying apply manifests before the webhook being ready can fail like that.

[andoriyu]

@andoriyu Unfortunately no. Your information indicates that everything's working well...

To be extra sure, have you tried holding on for a few seconds to a minute before applying example-org-runner.yaml? Depending on the environment, it can take some time until the admission webhook gets fully set up. And trying apply manifests before the webhook being ready can fail like that.

Yup, I tried waiting a whole day.

I don't really get why it's working fine without hooks installed and only fails when they are installed.

[mumoshu]

@andoriyu Out of curiosity, how is your K8s cluster provisioned? Self-managed, AWS EKS, or else? If you're on e.g. EKS, are you using non-standard CNI plugin like Cilium without the ENI support, Flannel, etc?

[mumoshu]

@andoriyu Oh, and I should have noted earlier... you can just delete mutatingwebhookconfiguration and validatingwebhookconfiguration without affecting actions-runner-controller. There's currently no specific feature that requires those admission webhooks. I just left it there thinking it wouldn't trap people like this!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[michaelst]

Can they be removed from the helm chart if they are not being used. They are also causing issues for me

[mumoshu]

@michaelst I have some plan to actually use it for more validation for usability.

Are you sure you've installed cert-manager beforehand? Reading all the issues and comments about this so far, I can't see why this happens at all if you have cert-manager correctly installed, you've cluster network correctly set up, and you have access to container image registry hosting images for cert-manager and actions-runner-controller and so on.

[mumoshu]

@andoriyu I recently learned about GKE a bit and your issue might be due your private GKE cluster's firewall blocking your K8s apiserver from accessing the webhook service. Does it work if you used port 443 instead of 8443 for your actions-runner-controller pods?(not the webhook service, as kube-proxy translates the k8s service access to pod access by iptables)

elastic/cloud-on-k8s#1437

[mumoshu]

Let me turn this into a discussion. Deprecating and removing the validating/mutating webhook isn't an option right now.

This still seems not an issue in actiosn-runner-controller code, but a cluster configuration issue. We need more information about your environment to further debug this, like what kind of cloud provider you use and how your cluster networking is configured.

[andoriyu]

@andoriyu I recently learned about GKE a bit and your issue might be due your private GKE cluster's firewall blocking your K8s apiserver from accessing the webhook service. Does it work if you used port 443 instead of 8443 for your actions-runner-controller pods?(not the webhook service, as kube-proxy translates the k8s service access to pod access by iptables)

elastic/cloud-on-k8s#1437

where can I change the port to try?

[mumoshu]

@andoriyu You might need to clone it as a local helm chart and change this line https://github.com/actions-runner-controller/actions-runner-controller/blob/082245c5db64e023cd79604d9b158f336770e3fe/charts/actions-runner-controller/templates/deployment.yaml#L73 and this line https://github.com/actions-runner-controller/actions-runner-controller/blob/082245c5db64e023cd79604d9b158f336770e3fe/charts/actions-runner-controller/templates/webhook_service.yaml#L12

[mumoshu]

@andoriyu Otherwise, I thought you can configure which port to open on your private GKE cluster and control-plane-to-data-plane?

[andoriyu]

@mumoshu I finally got to work on this again. So switching port "solved" the issue, but now it created a new one:

2021-07-22T00:54:14.205Z        ERROR   controller-runtime.controller   Reconciler error        {"controller": "runnerdeployment-controller", "request": "sre/dave-inc-runner", "error": "Internal error occurred: failed calling webhook \"mutate.runnerreplicaset.actions.summerwind.dev\": Post \"https://general-1-gh-runners-actions-runner-controller-webhook.gh-runners.svc:443/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset?timeout=30s\": x509: certificate is valid for general-1-gh-runners-actions-runner-controller-b8b49bd5-k84dx, not general-1-gh-runners-actions-runner-controller-webhook.gh-runners.svc"}
github.com/go-logr/zapr.(*zapLogger).Error

[Kampe]

Interesting, we're experiencing the same issue with private GKE clusters

[ys-am]

I'm sorry. I would like to ask for your support on this matter.
We are using AWS EKS, and the configuration of the NW is such that we have worker nodes in the privateSubnet and NatGateway in the publicSubnet.
What we have done is simple.
I installed cert-manager in helm.
After that, I followed the URL below to activate the Issuer.
https://cert-manager.io/docs/configuration/selfsigned/
Finally, I installed actions-runner-contoroller with kubectl, stored the Secret and confirmed that the controller-manager Pod is running.
Then, in the same way, "Error from server (InternalError): error when creating "repo-runner.yaml": Internal error occurred: failed calling webhook "mutate. runner.actions.summerwind.dev": Post "https://actions-runner-controller-webhook.actions-runner-system.svc:443/mutate-actions- The message "summerwind-dev-v1alpha1-runner?timeout=30s": service "actions-runner-controller-webhook" not found" is output.
This is the only error I'm getting and I'm honestly struggling to track it down.
Can you please support me?

[mumoshu]

@ys-am Hey! I hear you but, honestly speaking, I'm afraid AWS support would better help you. There're too many variables to cover and that might be too much for me... well, I do my best but you might see yourself soon that you'd better ask AWS support.

How's your security group settings for the control-plane and the nodes look like?

If you're using EKS, the control-plane needs to access "pods" behind the "k8s service" actions-runner-controller-webhook in the actions-runner-system namespace.

You don't need to allow the port :443, as it has nothing to do with security groups. kube-proxy on the control-plane should translate the access to svc:443 to $POD_IP:$PORT.

So your security group associated to the node must allow the control-plane security group to access :$PORT on the ENI attached the the pod in the node.

I thought EKS had support for assigning security groups dedicated for pod ENIs, not only inheriting the security group associated to the node to pods. If you're using e.g. Calico to control access to pods, that might also affect it. So, your mileage may vary.

Did it help?

[ys-am]

@mumoshu

Thank you.
That's what I meant, I'm working on EKS, no hindrance from NW-like places, that's not what I meant in my first explanation.

I'm wondering if Certmanager only needs to be installed, and if the service "actions-runner-controller-webhook" is really installed when installed with kubectl? I don't think I can find it in the kubectl stuff, so I'm not sure.

[shreyasGit]

I am facing exact same issue, was this solved and if yes how? can you post some details, iam installing it in EKS, it worked on a particular cluster, but failed for another error is - Error from server (InternalError): error when creating "self-hosted-runner-try.yaml": Internal error occurred: failed calling webhook "validate.runnerdeployment.actions.summerwind.dev": Post "https://actions-runner-controller-webhook.actions-runner-system.svc:443/validate-actions-summerwind-dev-v1alpha1-runnerdeployment?timeout=10s": context deadline exceeded
Any particular port is to be opened, i see 443 open in my case

[hawkesn]

Hi all, just for others that end up coming here from Google or GitHub. I managed to figure it out.

The Terraform EKS module (v.18+) had many breaking changes including locking down the security group to the cluster. If you are using this module, you will need to add this block:

node_security_group_additional_rules = {
    ingress_arc_webhook_tcp = {
      description                   = "Cluster to Actions Runner Controller webhook"
      protocol                      = "tcp"
      from_port                     = 9443
      to_port                       = 9443
      type                          = "ingress"
      source_cluster_security_group = true
    }
}

[ljobse]

The same problem seems to occur in GCP if autopilot is enabled.

[vinhnguyen-ct]

Thank you
I faced issue on GCP GKE and solved by opening flow 9443 from api (master) access to action runner controller application (pod networ)
vin

[samus-aran]

We're locking this discussion because it has not had recent activity and/or other members have asked for more information to assist you but received no response. Thank you for helping us maintain a productive and tidy community for all our members.