go tests sometimes may fail

[Ruminateer]

Go tests may fail with the following output. Seems to be a bug with test files but not sure yet.

$ make docker-build 
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/bin/controller-gen "crd:maxDescLen=0,trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
mkdir -p /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin
test -f /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh || curl -sSLo /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.0/hack/setup-envtest.sh
source /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh; fetch_envtest_tools /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin; setup_envtest_env /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin; go test ./... -coverprofile cover.out
/bin/sh: 1: source: not found
/bin/sh: 1: fetch_envtest_tools: not found
/bin/sh: 1: setup_envtest_env: not found
?       github.com/paddleflow/paddle-operator   [no test files]
?       github.com/paddleflow/paddle-operator/api/v1    [no test files]
Running Suite: Controller Suite
===============================
Random Seed: 1618384660
Will run 1 of 1 specs

• Failure [1.193 seconds]
PaddleJob controller
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/controllers/paddlejob_controller_test.go:18
  test wide-and-deep in service mode [It]
  /home/zzy/go/src/github.com/Ruminateer/paddle-operator/controllers/paddlejob_controller_test.go:98

  Expected success, but got an error:
      <*errors.StatusError | 0xc0003d75e0>: {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {
                  SelfLink: "",
                  ResourceVersion: "",
                  Continue: "",
                  RemainingItemCount: nil,
              },
              Status: "Failure",
              Message: "Operation cannot be fulfilled on paddlejobs.batch.paddlepaddle.org \"wide-and-deep-service\": the object has been modified; please apply your changes to the latest version and try again",
              Reason: "Conflict",
              Details: {
                  Name: "wide-and-deep-service",
                  Group: "batch.paddlepaddle.org",
                  Kind: "paddlejobs",
                  UID: "",
                  Causes: nil,
                  RetryAfterSeconds: 0,
              },
              Code: 409,
          },
      }
      Operation cannot be fulfilled on paddlejobs.batch.paddlepaddle.org "wide-and-deep-service": the object has been modified; please apply your changes to the latest version and try again

  /home/zzy/go/src/github.com/Ruminateer/paddle-operator/controllers/paddlejob_controller_test.go:94
------------------------------
2021-04-14T15:17:51.090+0800    INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2021-04-14T15:17:51.090+0800    INFO    controller-runtime.manager.controller.paddlejob Starting EventSource    {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob", "source": "kind source: /, Kind="}
2021-04-14T15:17:51.090+0800    INFO    controller-runtime.manager.controller.paddlejob Starting EventSource    {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob", "source": "kind source: /, Kind="}
2021-04-14T15:17:51.090+0800    INFO    controller-runtime.manager.controller.paddlejob Starting EventSource    {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob", "source": "kind source: /, Kind="}
2021-04-14T15:17:51.090+0800    INFO    controller-runtime.manager.controller.paddlejob Starting EventSource    {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob", "source": "kind source: /, Kind="}
2021-04-14T15:17:51.090+0800    INFO    controller-runtime.manager.controller.paddlejob Starting Controller     {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob"}
2021-04-14T15:17:51.090+0800    INFO    controller-runtime.manager.controller.paddlejob Starting workers        {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob", "worker count": 1}
2021-04-14T15:17:51.090+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "48"}
2021-04-14T15:17:51.104+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "50"}
2021-04-14T15:17:51.104+0800    DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"50"}, "reason": "Created", "message": "created  wide-and-deep-service-ps-0"}
2021-04-14T15:17:51.109+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "52"}
2021-04-14T15:17:51.109+0800    DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"52"}, "reason": "Created", "message": "created  wide-and-deep-service-ps-0"}
2021-04-14T15:17:51.111+0800    DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"52"}, "reason": "Created", "message": "created  wide-and-deep-service-ps-1"}
2021-04-14T15:17:51.111+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "52"}
2021-04-14T15:17:51.117+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "56"}
2021-04-14T15:17:51.117+0800    DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"56"}, "reason": "Created", "message": "created  wide-and-deep-service-ps-1"}
2021-04-14T15:17:51.121+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "56"}
2021-04-14T15:17:51.121+0800    DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"56"}, "reason": "Created", "message": "created  wide-and-deep-service-ps-2"}
2021-04-14T15:17:51.127+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "61"}
2021-04-14T15:17:51.127+0800    DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"61"}, "reason": "Created", "message": "created  wide-and-deep-service-ps-2"}
2021-04-14T15:17:52.132+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "65"}
2021-04-14T15:17:52.132+0800    DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"65"}, "reason": "Created", "message": "created  wide-and-deep-service-worker-0"}
2021-04-14T15:17:52.139+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "69"}
2021-04-14T15:17:52.139+0800    DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"69"}, "reason": "Created", "message": "created  wide-and-deep-service-worker-0"}
2021-04-14T15:17:52.142+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "69"}
2021-04-14T15:17:52.142+0800    DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"69"}, "reason": "Created", "message": "created  wide-and-deep-service-worker-1"}
2021-04-14T15:17:52.148+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "73"}
2021-04-14T15:17:52.149+0800    DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"73"}, "reason": "Created", "message": "created  wide-and-deep-service-worker-1"}
2021-04-14T15:17:52.166+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "77"}
2021-04-14T15:17:52.172+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "77"}
2021-04-14T15:17:52.179+0800    INFO    controllers.PaddleJob   Reconcile       {"paddlejob": "default/wide-and-deep-service", "version": "78"}



Summarizing 1 Failure:

[Fail] PaddleJob controller [It] test wide-and-deep in service mode 
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/controllers/paddlejob_controller_test.go:94

Ran 1 of 1 Specs in 11.493 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestAPIs (11.49s)
FAIL
coverage: 42.9% of statements
FAIL    github.com/paddleflow/paddle-operator/controllers       11.536s
FAIL
make: *** [Makefile:21: test] Error 1

This is not very reproducible. Most times it just succeeds.

$ make test
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/bin/controller-gen "crd:maxDescLen=0,trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
mkdir -p /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin
test -f /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh || curl -sSLo /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.0/hack/setup-envtest.sh
source /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh; fetch_envtest_tools /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin; setup_envtest_env /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin; go test ./... -coverprofile cover.out
/bin/sh: 1: source: not found
/bin/sh: 1: fetch_envtest_tools: not found
/bin/sh: 1: setup_envtest_env: not found
?       github.com/paddleflow/paddle-operator   [no test files]
?       github.com/paddleflow/paddle-operator/api/v1    [no test files]
ok      github.com/paddleflow/paddle-operator/controllers       6.290s  coverage: 47.8% of statements