Vagrant environment never started completely because of empty CLUSTERID #179
Description
Hello,
I tried to deploy local environment using Vagrant. The server creation went well but when it comes to the cluster creation, I kept getting failures using the original Vagrantfile and scripts.
Below is a chunk of the log :
server-01: ++ docker run --rm --net=host appropriate/curl -s 'https://127.0.0.1/v3-public/localProviders/local?action=login' -
H 'content-type: application/json' --data-binary '{"username":"admin","password":"admin"}' --insecure
server-01: + LOGINRESPONSE='{"authProvider":"local","baseType":"token","clusterId":null,"created":"2021-08-08T14:56:05Z","crea
tedTS":1628434565000,"creatorId":null,"current":false,"description":"","enabled":true,"expired":false,"expiresAt":"","groupPrincip
als":null,"id":"token-qm5m4","isDerived":false,"labels":{"authn.management.cattle.io/kind":"session","authn.management.cattle.io/t
oken-userId":"user-w84nz","cattle.io/creator":"norman"},"lastUpdateTime":"","links":{"self":"https://127.0.0.1/v3-public/tokens/to
ken-qm5m4"},"name":"token-qm5m4","token":"token-qm5m4:2z4529fbz6m9srzrt4chng94f6jgtlgnbfsxjp9bk45mcjkchwcf65","ttl":57600000,"type
":"token","userId":"user-w84nz","userPrincipal":"map[displayName:Default Admin loginName:admin me:true metadata:map[creationTimest
amp:\u003cnil\u003e name:local://user-w84nz] principalType:user provider:local]","uuid":"db74da64-60f3-4498-98d4-99c2ead9480b"}'
server-01: ++ docker run --rm -i stedolan/jq -r .token
server-01: ++ echo '{"authProvider":"local","baseType":"token","clusterId":null,"created":"2021-08-08T14:56:05Z","createdTS":1
628434565000,"creatorId":null,"current":false,"description":"","enabled":true,"expired":false,"expiresAt":"","groupPrincipals":nul
l,"id":"token-qm5m4","isDerived":false,"labels":{"authn.management.cattle.io/kind":"session","authn.management.cattle.io/token-use
rId":"user-w84nz","cattle.io/creator":"norman"},"lastUpdateTime":"","links":{"self":"https://127.0.0.1/v3-public/tokens/token-qm5m
4"},"name":"token-qm5m4","token":"token-qm5m4:2z4529fbz6m9srzrt4chng94f6jgtlgnbfsxjp9bk45mcjkchwcf65","ttl":57600000,"type":"token
","userId":"user-w84nz","userPrincipal":"map[displayName:Default' Admin loginName:admin me:true 'metadata:map[creationTimestamp:\u
003cnil\u003e' 'name:local://user-w84nz]' principalType:user 'provider:local]","uuid":"db74da64-60f3-4498-98d4-99c2ead9480b"}'
server-01: + LOGINTOKEN=token-qm5m4:2z4529fbz6m9srzrt4chng94f6jgtlgnbfsxjp9bk45mcjkchwcf65
server-01: + '[' token-qm5m4:2z4529fbz6m9srzrt4chng94f6jgtlgnbfsxjp9bk45mcjkchwcf65 '!=' null ']'
server-01: + break
server-01: + docker run --rm --net=host appropriate/curl -s 'https://127.0.0.1/v3/users?action=changepassword' -H 'content-typ
e: application/json' -H 'Authorization: Bearer token-qm5m4:2z4529fbz6m9srzrt4chng94f6jgtlgnbfsxjp9bk45mcjkchwcf65' --data-binary '
{"currentPassword":"admin","newPassword":"admin"}' --insecure
server-01: pong{"baseType":"error","code":"ServerError","message":"Put \"https://127.0.0.1:6443/apis/management.cattle.io/v3/s
ettings/first-login\": EOF","status":500,"type":"error"}
server-01: ++ docker run --rm --net=host appropriate/curl -s https://127.0.0.1/v3/token -H 'content-type: application/json' -H
'Authorization: Bearer token-qm5m4:2z4529fbz6m9srzrt4chng94f6jgtlgnbfsxjp9bk45mcjkchwcf65' --data-binary '{"type":"token","descri
ption":"automation"}' --insecure
server-01: + APIRESPONSE=
server-01: ++ docker run --rm -i stedolan/jq -r .token
server-01: ++ echo
server-01: + APITOKEN=
server-01: + RANCHER_SERVER=https://172.22.101.101
server-01: + docker run --rm --net=host appropriate/curl -s https://127.0.0.1/v3/settings/server-url -H 'content-type: applica
tion/json' -H 'Authorization: Bearer ' -X PUT --data-binary '{"name":"server-url","value":"https://172.22.101.101"}' --insecure
server-01: ++ docker run --rm --net=host appropriate/curl -s https://127.0.0.1/v3/cluster -H 'content-type: application/json'
-H 'Authorization: Bearer ' --data-binary '{"dockerRootDir":"/var/lib/docker","enableNetworkPolicy":false,"type":"cluster","ranche
rKubernetesEngineConfig":{"kubernetesVersion":"v1.18.15-rancher1-1","addonJobTimeout":30,"ignoreDockerVersion":true,"sshAgentAuth"
:false,"type":"rancherKubernetesEngineConfig","authentication":{"type":"authnConfig","strategy":"x509"},"network":{"options":{"fla
nnelBackendType":"vxlan"},"plugin":"canal","canalNetworkProvider":{"iface":"eth1"}},"ingress":{"type":"ingressConfig","provider":"
nginx"},"monitoring":{"type":"monitoringConfig","provider":"metrics-server"},"services":{"type":"rkeConfigServices","kubeApi":{"po
dSecurityPolicy":false,"type":"kubeAPIService"},"etcd":{"creation":"12h","extraArgs":{"heartbeat-interval":500,"election-timeout":
5000},"retention":"72h","snapshot":false,"type":"etcdService","backupConfig":{"enabled":true,"intervalHours":12,"retention":6,"typ
e":"backupConfig"}}}},"localClusterAuthEndpoint":{"enabled":true,"type":"localClusterAuthEndpoint"},"name":"quickstart"}' --insecu
re
server-01: + CLUSTERRESPONSE=
server-01: ++ echo
server-01: ++ docker run --rm -i stedolan/jq -r .id
server-01: + CLUSTERID=
server-01: + docker run --rm --net=host appropriate/curl -s https://127.0.0.1/v3/clusterregistrationtoken -H 'content-type: ap
plication/json' -H 'Authorization: Bearer ' --data-binary '{"type":"clusterRegistrationToken","clusterId":""}' --insecure
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.
As you can see, CLUSTERRESPONSE and CLUSTERID are empty because the returned answers to the corresponding docker run curl commands are not as expected (mainly because TOKENID is empty).
This failure comes normally because the curl commands are executed too early and the server are not ready yet.
So, a bit of modification within configure_rancher_server.sh can handle this. We need to retry the curl commands to create the cluster. My trial showed that at the second time the cluster creation worked !
I'll send a PR for this.