Resolved CrashLoopBackOff Issue (driver = docker)

[asusikai]

What Happened?

I was trying to use Jenkins with Helm, but the pod kept entering a CrashLoopBackOff state. The issue was resolved by changing the permissions of the directory /tmp/hostpath-provisioner/default/[helm-chart-dir] on the pod from 755 to 777.

I tried this on both Mac and Windows.

I believe this issue could be resolved by modifying the directory creation permissions from 0755 to 0777 in the following file:

https://github.com/kubernetes/minikube/blob/master/cmd/storage-provisioner/main.go

Attach the log file

dongjun@andongjun-ui-MacBookAir ~ % helm repo add jenkinsci https://charts.jenkins.io/
"jenkinsci" has been added to your repositories
dongjun@andongjun-ui-MacBookAir ~ % helm install my-jenkins jenkinsci/jenkins --version 5.5.6
NAME: my-jenkins
LAST DEPLOYED: Mon Aug 19 16:29:37 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:

1. Get your 'admin' user password by running:
   kubectl exec --namespace default -it svc/my-jenkins -c jenkins -- /bin/cat /run/secrets/additional/chart-admin-password && echo

2. Get the Jenkins URL to visit by running these commands in the same shell:
   echo http://127.0.0.1:8080
   kubectl --namespace default port-forward svc/my-jenkins 8080:8080

3. Login with the password from step 1 and the username: admin

4. Configure security realm and authorization strategy

5. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: http://127.0.0.1:8080/configuration-as-code and examples: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos

For more information on running Jenkins on Kubernetes, visit:
https://cloud.google.com/solutions/jenkins-on-container-engine

For more information about Jenkins Configuration as Code, visit:
https://jenkins.io/projects/jcasc/

NOTE: Consider using a custom image with pre-installed plugins
dongjun@andongjun-ui-MacBookAir ~ % kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-test Ready control-plane 15m v1.28.1
kube-test-m02 Ready 14m v1.28.1
kube-test-m03 Ready 14m v1.28.1
kube-test-m04 Ready 14m v1.28.1
dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods
NAME READY STATUS RESTARTS AGE
my-jenkins-0 0/2 Init:1/2 0 9s
dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods
NAME READY STATUS RESTARTS AGE
my-jenkins-0 0/2 Init:1/2 0 11s
dongjun@andongjun-ui-MacBookAir ~ % watch -n1 kubectl get pods
zsh: command not found: watch
dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods
NAME READY STATUS RESTARTS AGE
my-jenkins-0 0/2 Init:CrashLoopBackOff 1 (2s ago) 23s
dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods
NAME READY STATUS RESTARTS AGE
my-jenkins-0 0/2 Init:CrashLoopBackOff 1 (4s ago) 25s
dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods
NAME READY STATUS RESTARTS AGE
my-jenkins-0 0/2 Init:CrashLoopBackOff 1 (7s ago) 28s
dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods
NAME READY STATUS RESTARTS AGE
my-jenkins-0 0/2 Init:CrashLoopBackOff 4 (61s ago) 2m54s
dongjun@andongjun-ui-MacBookAir ~ % kubectl describe pod my-jenikns-0
Error from server (NotFound): pods "my-jenikns-0" not found
dongjun@andongjun-ui-MacBookAir ~ % kubectl describe pod my-jenkins-0
Name: my-jenkins-0
Namespace: default
Priority: 0
Service Account: my-jenkins
Node: kube-test-m04/192.168.49.5
Start Time: Mon, 19 Aug 2024 16:29:37 +0900
Labels: app.kubernetes.io/component=jenkins-controller
app.kubernetes.io/instance=my-jenkins
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=jenkins
apps.kubernetes.io/pod-index=0
controller-revision-hash=my-jenkins-7dd4db4cd5
statefulset.kubernetes.io/pod-name=my-jenkins-0
Annotations: checksum/config: bf5c41547ad32f6807e04f82058cd4540d545e6b947509422beff6f7e4491ff0
Status: Pending
IP: 10.244.3.3
IPs:
IP: 10.244.3.3
Controlled By: StatefulSet/my-jenkins
Init Containers:
config-reload-init:
Container ID: docker://964d6d4718be0f8c99663902e8f481905c40fb491d1aff83964784de137f2484
Image: docker.io/kiwigrid/k8s-sidecar:1.27.5
Image ID: docker-pullable://kiwigrid/k8s-sidecar@sha256:1fc88232e223a6977c626510372a74ca1f73af073e3c361719ccf02f223c8a12
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 19 Aug 2024 16:29:43 +0900
Finished: Mon, 19 Aug 2024 16:29:45 +0900
Ready: True
Restart Count: 0
Environment:
POD_NAME: my-jenkins-0 (v1:metadata.name)
LABEL: my-jenkins-jenkins-config
FOLDER: /var/jenkins_home/casc_configs
NAMESPACE: default
METHOD: LIST
Mounts:
/var/jenkins_home from jenkins-home (rw)
/var/jenkins_home/casc_configs from sc-config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ckzfj (ro)
init:
Container ID: docker://cfbfab6b35185d24a1934d95d65c6634046ebd06a30ae28c340a934ad2db4a43
Image: docker.io/jenkins/jenkins:2.462.1-jdk17
Image ID: docker-pullable://jenkins/jenkins@sha256:2bafb1fb2d6489bccadc1b7c172937e9b56a888ed77e625a4ebe59a6b038221e
Port:
Host Port:
Command:
sh
/var/jenkins_config/apply_config.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Mon, 19 Aug 2024 16:37:13 +0900
Finished: Mon, 19 Aug 2024 16:37:13 +0900
Ready: False
Restart Count: 5
Limits:
cpu: 2
memory: 4Gi
Requests:
cpu: 50m
memory: 256Mi
Environment:
Mounts:
/tmp from tmp-volume (rw)
/usr/share/jenkins/ref/plugins from plugins (rw)
/var/jenkins_config from jenkins-config (rw)
/var/jenkins_home from jenkins-home (rw)
/var/jenkins_plugins from plugin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ckzfj (ro)
Containers:
jenkins:
Container ID:
Image: docker.io/jenkins/jenkins:2.462.1-jdk17
Image ID:
Ports: 8080/TCP, 50000/TCP
Host Ports: 0/TCP, 0/TCP
Args:
--httpPort=8080
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 4Gi
Requests:
cpu: 50m
memory: 256Mi
Liveness: http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=3
Startup: http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=12
Environment:
SECRETS: /run/secrets/additional
POD_NAME: my-jenkins-0 (v1:metadata.name)
JAVA_OPTS: -Dcasc.reload.token=$(POD_NAME)
JENKINS_OPTS: --webroot=/var/jenkins_cache/war
JENKINS_SLAVE_AGENT_PORT: 50000
CASC_JENKINS_CONFIG: /var/jenkins_home/casc_configs
Mounts:
/run/secrets/additional from jenkins-secrets (ro)
/tmp from tmp-volume (rw)
/usr/share/jenkins/ref/plugins/ from plugin-dir (rw)
/var/jenkins_cache from jenkins-cache (rw)
/var/jenkins_config from jenkins-config (ro)
/var/jenkins_home from jenkins-home (rw)
/var/jenkins_home/casc_configs from sc-config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ckzfj (ro)
config-reload:
Container ID:
Image: docker.io/kiwigrid/k8s-sidecar:1.27.5
Image ID:
Port:
Host Port:
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
POD_NAME: my-jenkins-0 (v1:metadata.name)
LABEL: my-jenkins-jenkins-config
FOLDER: /var/jenkins_home/casc_configs
NAMESPACE: default
REQ_URL: http://localhost:8080/reload-configuration-as-code/?casc-reload-token=$(POD_NAME)
REQ_METHOD: POST
REQ_RETRY_CONNECT: 10
Mounts:
/var/jenkins_home from jenkins-home (rw)
/var/jenkins_home/casc_configs from sc-config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ckzfj (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
plugins:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
jenkins-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: my-jenkins
Optional: false
plugin-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
jenkins-secrets:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: my-jenkins
SecretOptionalName:
jenkins-cache:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
jenkins-home:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: my-jenkins
ReadOnly: false
sc-config-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
tmp-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
kube-api-access-ckzfj:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 7m50s default-scheduler Successfully assigned default/my-jenkins-0 to kube-test-m04
Normal Pulling 7m49s kubelet Pulling image "docker.io/kiwigrid/k8s-sidecar:1.27.5"
Normal Pulled 7m44s kubelet Successfully pulled image "docker.io/kiwigrid/k8s-sidecar:1.27.5" in 5.707s (5.707s including waiting)
Normal Created 7m44s kubelet Created container config-reload-init
Normal Started 7m44s kubelet Started container config-reload-init
Normal Pulled 7m32s kubelet Successfully pulled image "docker.io/jenkins/jenkins:2.462.1-jdk17" in 10.113s (10.113s including waiting)
Normal Pulled 7m29s kubelet Successfully pulled image "docker.io/jenkins/jenkins:2.462.1-jdk17" in 1.827s (1.827s including waiting)
Normal Pulled 7m11s kubelet Successfully pulled image "docker.io/jenkins/jenkins:2.462.1-jdk17" in 1.898s (1.898s including waiting)
Normal Pulling 6m42s (x4 over 7m42s) kubelet Pulling image "docker.io/jenkins/jenkins:2.462.1-jdk17"
Normal Created 6m41s (x4 over 7m32s) kubelet Created container init
Normal Pulled 6m41s kubelet Successfully pulled image "docker.io/jenkins/jenkins:2.462.1-jdk17" in 1.798s (1.798s including waiting)
Normal Started 6m40s (x4 over 7m32s) kubelet Started container init
Warning BackOff 6m40s (x5 over 7m29s) kubelet Back-off restarting failed container init in pod my-jenkins-0_default(d7efc991-9c1e-4321-ad3b-8a88fa283b08)

Operating System

macOS (Default)

Driver

Docker

[asusikai]

@gnufied could you please add ev/CrashLoopBackoff label to this issue? thx

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.