Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]Etcd cluster is always creating #7935

Closed
haowen159 opened this issue Aug 7, 2024 · 3 comments
Closed

[BUG]Etcd cluster is always creating #7935

haowen159 opened this issue Aug 7, 2024 · 3 comments
Assignees
@free6om @loomts
Labels
kind/bug Something isn't working
Milestone
Release 0.9.2

Comments

@haowen159
Copy link
Contributor

Describe the bug
A clear and concise description of what the bug is.

kbcli version
Kubernetes: v1.28.11-gke.1019001
KubeBlocks: 1.0.0-alpha.1
kbcli: 1.0.0-alpha.0

To Reproduce
Steps to reproduce the behavior:

  1. cluster yaml
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: etcd-cluster
  namespace: default
spec:
  terminationPolicy: Delete
  componentSpecs:
  - name: etcd
    componentDef: etcd-1.0.0-alpha.0
    serviceAccountName: kb-etcd-cluster
    replicas: 3
    resources:
      limits:
        cpu: '0.5'
        memory: 0.5Gi
      requests:
        cpu: '0.5'
        memory: 0.5Gi
    volumeClaimTemplates:
    - name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi

2.cluster status:

 k get cluster
NAME           CLUSTER-DEFINITION   VERSION   TERMINATION-POLICY   STATUS     AGE
etcd-cluster                                  Delete               Creating   13m
k get pod -l app.kubernetes.io/instance=etcd-cluster
NAME                  READY   STATUS    RESTARTS   AGE
etcd-cluster-etcd-0   2/2     Running   0          13m
etcd-cluster-etcd-1   2/2     Running   0          13m
etcd-cluster-etcd-2   2/2     Running   0          13m
 k get cmp -l app.kubernetes.io/instance=etcd-cluster
NAME                DEFINITION           SERVICE-VERSION   STATUS     AGE
etcd-cluster-etcd   etcd-1.0.0-alpha.0   v3.5.6            Creating   14m

3.logs

k logs etcd-cluster-etcd-2 -c lorry
2024-08-07T02:25:24Z    INFO    Initialize DB manager
2024-08-07T02:25:24Z    INFO    KB_WORKLOAD_TYPE ENV not set
2024-08-07T02:25:24Z    INFO    Volume-Protection       succeed to init volume protection       {"pod": "etcd-cluster-etcd-2", "spec": {"highWatermark":"0","volumes":[]}}
2024-08-07T02:25:24Z    INFO    HTTPServer      Starting HTTP Server
2024-08-07T02:25:24Z    INFO    HTTPServer      API route path  {"method": "POST", "path": ["/v1.0/joinmember", "/v1.0/createuser", "/v1.0/checkrunning", "/v1.0/leavemember", "/v1.0/lockinstance", "/v1.0/exec", "/v1.0/switchover", "/v1.0/dataload", "/v1.0/volumeprotection", "/v1.0/postprovision", "/v1.0/rebuild", "/v1.0/grantuserrole", "/v1.0/revokeuserrole", "/v1.0/preterminate", "/v1.0/datadump", "/v1.0/getlag", "/v1.0/deleteuser", "/v1.0/unlockinstance"]}
2024-08-07T02:25:24Z    INFO    HTTPServer      API route path  {"method": "GET", "path": ["/v1.0/healthycheck", "/v1.0/query", "/v1.0/getrole", "/v1.0/listusers", "/v1.0/listsystemaccounts", "/v1.0/checkrole", "/v1.0/describeuser"]}
2024-08-07T02:25:24Z    INFO    cronjobs        env is not set  {"env": "KB_CRON_JOBS"}
2024-08-07T02:25:30Z    INFO    DCS-K8S pod selector: app.kubernetes.io/instance=etcd-cluster,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=etcd
2024-08-07T02:25:30Z    INFO    DCS-K8S podlist: 3
2024-08-07T02:25:30Z    INFO    DCS-K8S Leader configmap is not found   {"configmap": "etcd-cluster-etcd-leader"}
2024-08-07T02:25:30Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:25:30Z    INFO    checkrole       role checks failed continuously {"times": 0}
2024-08-07T02:25:30Z    INFO    event   send event: map[operation:checkRole originalRole:waitForStart]
2024-08-07T02:25:31Z    INFO    event   send event success      {"message": "{\"operation\":\"checkRole\",\"originalRole\":\"waitForStart\"}"}
2024-08-07T02:25:40Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:25:50Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:26:00Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:26:10Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:26:20Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:26:30Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:26:40Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:26:50Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:27:00Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:27:10Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:27:20Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:27:30Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:27:40Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:27:50Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:28:00Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:28:10Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:28:20Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:28:30Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:28:40Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:28:50Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:29:00Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:29:10Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:29:20Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:29:30Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:29:40Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:29:50Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T02:30:00Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@haowen159 haowen159 added the kind/bug Something isn't working label Aug 7, 2024
@haowen159
Copy link
Contributor Author

The Zookeeper cluster also has the same bug

k get cluster zkeeper-iyrxre            
NAME             CLUSTER-DEFINITION   VERSION   TERMINATION-POLICY   STATUS     AGE
zkeeper-iyrxre                                  Delete               Creating   5m41s
k get pod
NAME                         READY   STATUS    RESTARTS   AGE
minio-6d88f87459-ttgx5       1/1     Running   0          25m
zkeeper-iyrxre-zookeeper-0   2/2     Running   0          5m54s

 k logs zkeeper-iyrxre-zookeeper-0 -c lorry
2024-08-07T07:32:31Z    INFO    Initialize DB manager
2024-08-07T07:32:31Z    INFO    KB_WORKLOAD_TYPE ENV not set
2024-08-07T07:32:31Z    INFO    Volume-Protection       succeed to init volume protection       {"pod": "zkeeper-iyrxre-zookeeper-0", "spec": {"highWatermark":"0","volumes":[]}}
2024-08-07T07:32:31Z    INFO    HTTPServer      Starting HTTP Server
2024-08-07T07:32:31Z    INFO    HTTPServer      API route path  {"method": "POST", "path": ["/v1.0/checkrunning", "/v1.0/leavemember", "/v1.0/switchover", "/v1.0/grantuserrole", "/v1.0/preterminate", "/v1.0/createuser", "/v1.0/getlag", "/v1.0/exec", "/v1.0/deleteuser", "/v1.0/postprovision", "/v1.0/joinmember", "/v1.0/lockinstance", "/v1.0/volumeprotection", "/v1.0/unlockinstance", "/v1.0/rebuild", "/v1.0/revokeuserrole", "/v1.0/datadump", "/v1.0/dataload"]}
2024-08-07T07:32:31Z    INFO    HTTPServer      API route path  {"method": "GET", "path": ["/v1.0/listsystemaccounts", "/v1.0/listusers", "/v1.0/describeuser", "/v1.0/getrole", "/v1.0/query", "/v1.0/checkrole", "/v1.0/healthycheck"]}
2024-08-07T07:32:31Z    INFO    cronjobs        env is not set  {"env": "KB_CRON_JOBS"}
2024-08-07T07:32:35Z    INFO    DCS-K8S pod selector: app.kubernetes.io/instance=zkeeper-iyrxre,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=zookeeper
2024-08-07T07:32:35Z    INFO    DCS-K8S podlist: 1
2024-08-07T07:32:35Z    INFO    DCS-K8S Leader configmap is not found   {"configmap": "zkeeper-iyrxre-zookeeper-leader"}
2024-08-07T07:32:35Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T07:32:35Z    INFO    checkrole       role checks failed continuously {"times": 0}
2024-08-07T07:32:35Z    INFO    event   send event: map[operation:checkRole originalRole:waitForStart]
2024-08-07T07:32:35Z    INFO    event   send event success      {"message": "{\"operation\":\"checkRole\",\"originalRole\":\"waitForStart\"}"}
2024-08-07T07:32:44Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T07:32:54Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T07:33:04Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T07:33:14Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T07:33:24Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T07:33:34Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T07:33:44Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}
2024-08-07T07:33:54Z    INFO    checkrole       executing checkRole error       {"error": "role probe commands is empty!"}

@loomts
Copy link
Contributor

loomts commented Aug 26, 2024

@haowen159 the role probe action is refactoring by kbagent, please wait~

@haowen159
Copy link
Contributor Author

The bug is fixed.

@github-actions github-actions bot added this to the Release 0.9.2 milestone Aug 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@free6om free6om

@loomts loomts

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.9.2
Development

No branches or pull requests
3 participants
@free6om @loomts @haowen159