Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]The role of mysql pod changed from follower to none after hscale 3-1 #4959

Closed
ahjing99 opened this issue Sep 1, 2023 · 1 comment
Closed

[BUG]The role of mysql pod changed from follower to none after hscale 3-1 #4959

ahjing99 opened this issue Sep 1, 2023 · 1 comment
Assignees
@free6om
Labels
bug kind/bug Something isn't working
Milestone
Release 0.7.0

Comments

@ahjing99
Copy link
Collaborator

ahjing99 commented Sep 1, 2023

➜ ~ kbcli version
Kubernetes: v1.27.3-gke.100
KubeBlocks: 0.7.0-alpha.4
kbcli: 0.7.0-alpha.4

  1. Create cluster with 1 replica
➜  ~ kbcli cluster create  mysqltest2             --termination-policy=DoNotTerminate             --monitoring-interval=0 --enable-all-logs=false --cluster-definition=apecloud-mysql --cluster-version=ac-mysql-8.0.30 --set cpu=100m,memory=0.5Gi,replicas=1,storage=4Gi
Cluster mysqltest2 created

➜  ~ kbcli cluster describe mysqltest2
Name: mysqltest2	 Created Time: Sep 01,2023 14:43 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS    TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30   Running   DoNotTerminate

Endpoints:
COMPONENT   MODE        INTERNAL                                          EXTERNAL
mysql       ReadWrite   mysqltest2-mysql.default.svc.cluster.local:3306   <none>

Topology:
COMPONENT   INSTANCE             ROLE     STATUS    AZ              NODE                                                CREATED-TIME
mysql       mysqltest2-mysql-0   leader   Running   us-central1-c   gke-yjtest-default-pool-f59be211-2vqs/10.128.0.46   Sep 01,2023 14:43 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mysql       false       100m / 100m          512Mi / 512Mi           data:4Gi       kb-default-sc

Images:
COMPONENT   TYPE    IMAGE
mysql       mysql   registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13

Data Protection:
AUTO-BACKUP   BACKUP-SCHEDULE   TYPE     BACKUP-TTL   LAST-SCHEDULE   RECOVERABLE-TIME
Disabled      <none>            <none>   7d           <none>          <none>

Show cluster events: kbcli cluster list-events -n default mysqltest2
  1. Hscale 1-3
➜  ~ kbcli cluster hscale mysqltest2 --auto-approve --components mysql --replicas 3
OpsRequest mysqltest2-horizontalscaling-kf9lc created successfully, you can view the progress:
	kbcli cluster describe-ops mysqltest2-horizontalscaling-kf9lc -n default

➜  ~ kbcli cluster describe mysqltest2
Name: mysqltest2	 Created Time: Sep 01,2023 14:43 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS    TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30   Running   DoNotTerminate

Endpoints:
COMPONENT   MODE        INTERNAL                                          EXTERNAL
mysql       ReadWrite   mysqltest2-mysql.default.svc.cluster.local:3306   <none>

Topology:
COMPONENT   INSTANCE             ROLE       STATUS    AZ              NODE                                                CREATED-TIME
mysql       mysqltest2-mysql-0   leader     Running   us-central1-c   gke-yjtest-default-pool-f59be211-2vqs/10.128.0.46   Sep 01,2023 14:43 UTC+0800
mysql       mysqltest2-mysql-1   follower   Running   us-central1-c   gke-yjtest-default-pool-f59be211-53bj/10.128.0.47   Sep 01,2023 14:47 UTC+0800
mysql       mysqltest2-mysql-2   follower   Running   us-central1-c   gke-yjtest-default-pool-f59be211-1jst/10.128.0.45   Sep 01,2023 14:47 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mysql       false       100m / 100m          512Mi / 512Mi           data:4Gi       kb-default-sc

Images:
COMPONENT   TYPE    IMAGE
mysql       mysql   registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13

Data Protection:
AUTO-BACKUP   BACKUP-SCHEDULE   TYPE     BACKUP-TTL   LAST-SCHEDULE   RECOVERABLE-TIME
Disabled      <none>            <none>   7d           <none>          <none>

Show cluster events: kbcli cluster list-events -n default mysqltest2
  1. Hscale 3-1
➜  ~ kbcli cluster hscale mysqltest2 --auto-approve --components mysql --replicas 1
OpsRequest mysqltest2-horizontalscaling-c89mp created successfully, you can view the progress:
	kbcli cluster describe-ops mysqltest2-horizontalscaling-c89mp -n default

➜  ~ kbcli cluster describe-ops mysqltest2-horizontalscaling-c89mp -n default
Spec:
  Name: mysqltest2-horizontalscaling-c89mp	NameSpace: default	Cluster: mysqltest2	Type: HorizontalScaling

Command:
  kbcli cluster hscale mysqltest2 --components=mysql --replicas=1 --namespace=default

Last Configuration:
COMPONENT   REPLICAS
mysql              3

Status:
  Start Time:         Sep 01,2023 14:50 UTC+0800
  Completion Time:    Sep 01,2023 14:50 UTC+0800
  Duration:           35s
  Status:             Succeed
  Progress:           2/2
                      OBJECT-KEY               STATUS    DURATION   MESSAGE
                      Pod/mysqltest2-mysql-2   Succeed   32s        Successfully delete pod: Pod/mysqltest2-mysql-2 in Component: mysql
                      Pod/mysqltest2-mysql-1   Succeed   32s        Successfully delete pod: Pod/mysqltest2-mysql-1 in Component: mysql

Conditions:
LAST-TRANSITION-TIME         TYPE                REASON                            STATUS   MESSAGE
Sep 01,2023 14:50 UTC+0800   Progressing         OpsRequestProgressingStarted      True     Start to process the OpsRequest: mysqltest2-horizontalscaling-c89mp in Cluster: mysqltest2
Sep 01,2023 14:50 UTC+0800   Validated           ValidateOpsRequestPassed          True     OpsRequest: mysqltest2-horizontalscaling-c89mp is validated
Sep 01,2023 14:50 UTC+0800   HorizontalScaling   HorizontalScalingStarted          True     Start to horizontal scale replicas in Cluster: mysqltest2
Sep 01,2023 14:50 UTC+0800   Succeed             OpsRequestProcessedSuccessfully   True     Successfully processed the OpsRequest: mysqltest2-horizontalscaling-c89mp in Cluster: mysqltest2

Warning Events: <none>

➜  ~ kbcli cluster describe mysqltest2
Name: mysqltest2	 Created Time: Sep 01,2023 14:43 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS    TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30   Running   DoNotTerminate

Endpoints:
COMPONENT   MODE        INTERNAL                                          EXTERNAL
mysql       ReadWrite   mysqltest2-mysql.default.svc.cluster.local:3306   <none>

Topology:
COMPONENT   INSTANCE             ROLE       STATUS    AZ              NODE                                                CREATED-TIME
mysql       mysqltest2-mysql-0   follower   Running   us-central1-c   gke-yjtest-default-pool-f59be211-2vqs/10.128.0.46   Sep 01,2023 14:43 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mysql       false       100m / 100m          512Mi / 512Mi           data:4Gi       kb-default-sc

Images:
COMPONENT   TYPE    IMAGE
mysql       mysql   registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13

Data Protection:
AUTO-BACKUP   BACKUP-SCHEDULE   TYPE     BACKUP-TTL   LAST-SCHEDULE   RECOVERABLE-TIME
Disabled      <none>            <none>   7d           <none>          <none>

Show cluster events: kbcli cluster list-events -n default mysqltest2
  1. After a while, the role changed from follower to none
➜  ~ kbcli cluster describe mysqltest2
Name: mysqltest2	 Created Time: Sep 01,2023 14:43 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS    TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30   Running   DoNotTerminate

Endpoints:
COMPONENT   MODE        INTERNAL                                          EXTERNAL
mysql       ReadWrite   mysqltest2-mysql.default.svc.cluster.local:3306   <none>

Topology:
COMPONENT   INSTANCE             ROLE     STATUS    AZ              NODE                                                CREATED-TIME
mysql       mysqltest2-mysql-0   <none>   Running   us-central1-c   gke-yjtest-default-pool-f59be211-2vqs/10.128.0.46   Sep 01,2023 14:43 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mysql       false       100m / 100m          512Mi / 512Mi           data:4Gi       kb-default-sc

Images:
COMPONENT   TYPE    IMAGE
mysql       mysql   registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13

Data Protection:
AUTO-BACKUP   BACKUP-SCHEDULE   TYPE     BACKUP-TTL   LAST-SCHEDULE   RECOVERABLE-TIME
Disabled      <none>            <none>   7d           <none>          <none>

Show cluster events: kbcli cluster list-events -n default mysqltest2
@ahjing99 ahjing99 added the kind/bug Something isn't working label Sep 1, 2023
@ahjing99 ahjing99 added this to the Release 0.7.0 milestone Sep 1, 2023
@ahjing99 ahjing99 mentioned this issue Sep 1, 2023
Closed
@free6om
Copy link
Contributor

free6om commented Sep 27, 2023

fixed in #4835

@free6om free6om closed this as completed Sep 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@free6om free6om

Labels
bug kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.7.0
Development

No branches or pull requests
3 participants
@free6om @ahjing99 @apecloud-bot