skaffold verify should support env vars with templating

[nathanperkins]

The skaffold.yaml API for verify container only documents usage for name, image, command and args. This is insufficient for us to specify our tests which need env vars and need to be able to template values based on the environment that is running skaffold, similar to what we can do for our deployed helm charts.

We would like to do something like:

verify:
  container:
  - name: foo
    env:
    - name: MY_STATIC_ENV
       value: "Hello from the environment"
    - name: MY_TEMPLATE_ENV
       value: "Hello from {{.MY_EXTERNAL_VAR}}"
    - name: MY_TEST_IMAGE
       value: "{{.IMAGE_FULLY_QUALIFIED_my-test-image}}"

Information

• Skaffold version: 2.13.0
• Operating system: Linux
• Installed via: skaffold.dev
• Contents of skaffold.yaml: see above

[nathanperkins]

I checked whether it is possible to achieve the desired result using overrides but it doesn't seem so. It doesn't seem to be a strategic merge so the image tag is lost when your override specifies the container. This means you can't make any overrides to a container?

verify:
  - name: foo-e2e-test
    container:
      name: foo-e2e-test
      image: foo-e2e-test
      args:
      - -test.v
    executionMode:
      kubernetesCluster:
        overrides: |-
          {
            "kind": "Job",
            "apiVersion": "batch/v1",
            "spec": {
              "template": {
                "spec": {
                  "containers": [
                    {
                      "name": "foo-e2e-test",
                      "env": [
                        {
                          "name": "FOO",
                          "value": "BAR"
                        }
                      ]
                    }
                  ]
                }
              }
            }
          }

DEBU[0065] Running command: [tput colors]                subtask=-1 task=DevLoop
DEBU[0065] Command output: [256
]                        subtask=-1 task=DevLoop
1 error(s) occurred:
* creating verify job in cluster: Job.batch "foo-e2e-test" is invalid: spec.template.spec.containers[0].image: Required value

[nathanperkins]

Tried this too, but it doesn't resolve the templated image within the overrides:

verify:
  - name: foo-e2e-test
    container:
      name: foo-e2e-test
      image: foo-e2e-test
    executionMode:
      kubernetesCluster:
        overrides: |-
          {
            "kind": "Job",
            "apiVersion": "batch/v1",
            "spec": {
              "template": {
                "spec": {
                  "containers": [
                    {
                      "name": "foo-e2e-test",
                      "image": "{{.IMAGE_FULLY_QUALIFIED_foo-e2e-test}}",
                      "args": [
                        "-test.v"
                      ],
                      "env": [
                        {
                          "name": "FOO",
                          "value": "BAR"
                        }
                      ]
                    }
                  ]
                }
              }
            }
          }

[nathanperkins]

It seems that there is a --env-file flag but that's pretty uncomfortable for us to use. We went all in on using env vars profiles in skaffold.yaml to parameterize our engineer, staging, and prod environments. For engineer environments (unique to each engineer), the dynamic values are provided via env var templating. For staging and prod environments, they are hard-coded into the skaffold.yaml.

For an E2E test, there are dynamic values that we need to be able to pass to the verify container. We are going to be deploying and running the verify as an E2E test so we want to be able to easily change the namespace, provide a unique identifier for the test run, etc. We could burn these values into an env-file but then we can't use the normal overlapping profiles with overrides strategy that we've been using successfully for skaffold so far.

Maybe some more complete examples for complex use cases of skaffold verify would help as well?

[nathanperkins]

Another example of a dynamic value we need to pass to the E2E test: we're testing a system which needs to create pods whose containers use images built by skaffold so we need to be able to tell the test the tag for those images. It's usually possible to reference these with {{.IMAGE_FULLY_QUALIFIED_foo-e2e-test}}. Without the ability to template these as part of dynamic values, our scripts will have to run skaffold build to output the artifacts as a JSON file, then use jq to extract, then write to a dotenv file, then use that dotenv file to run our verifications.