updates to ImageConfig do not propagate to STS

[Miles-Garnsey]

What happened?

There are unexpected behaviours when the ImageConfig ConfigMap is updated at runtime. For example:

1. The ConfigMap is not polled/watched by the operator, which means the operator's pod must be restarted to update the ImageConfig, despite the fact that the file has been updated on the filesystem.
2. Even once the ImageConfig updates have passed into the operator, the Cassandra StatefulSets are not updated.

Did you expect to see something different?

Yes. As soon as the ImageConfig file changes on the filesystem I expected the Statefulset to be updated and undergo a rolling restart.

How to reproduce it (as minimally and precisely as possible):

1. Create a new cass-operator deployment.
2. Deploy a CassandraDatacenter.
3. Update the ImageConfig (see the below kustomization for an example of how to do so).
4. You will observe that the statefulset is not updated.
5. Create a new CassandraDatacenter. It will still be using the old image. Delete it.
6. Restart the operator.
7. Observe that the CassandraDatacenter is still using the old image and Statefulsets have not restarted.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../config/deployments/cluster

patches:
- target:
    kind: ConfigMap
    name: cass-operator-manager-config
  patch: |
    - op: replace 
      path: /data/image_config.yaml
      value: |
        apiVersion: config.k8ssandra.io/v1beta1
        kind: ImageConfig
        metadata:
          name: image-config
        images:
          system-logger: "my-registry/system-logger-image-test:v0.0.1"
          config-builder: "datastax/cass-config-builder:1.0.4-ubi7"
          # cassandra:
          #   "4.0.0": "k8ssandra/cassandra-ubi:latest"
          # dse:
          #   "6.8.999": "datastax/dse-server-prototype:latest"
        # imageRegistry: "localhost:5000"
        # imagePullPolicy: Always
        # imagePullSecret:
        #   name: my-secret-pull-registry
        defaults:
          # Note, postfix is ignored if repository is not set
          cassandra:
            repository: "k8ssandra/cass-management-api"
          dse:
            repository: "datastax/dse-server"
            suffix: "-ubi7"

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: CASS-47

[jsanda]

I do not agree with what is being proposed for expected behavior. I say that because the ImageConfig is not part of the CassandraDatacenter spec. It's just used for applying defaults. Consider this scenario. The operator is deployed by an admin. I have perms to deploy CassandraDataCenters in a particular namespace. I have one up and running for a while. The admin changes the ImageConfig map and my DC gets updated. I wouldn't be very happy about that :)

[Miles-Garnsey]

@jsanda I think following discussions today (documented in Slack, but good to also put them here) I am recommending:

1. The ImageConfig is indeed an 'admin/default images' type of object and probably shouldn't be set by users in the normal course of operations.
2. Users should be able to set images for all cass-operator containers in a uniform and standardised way across both cass-operator and k8ssandra-operator.
3. ImageConfig should not be used to pass images down from k8ssandra-operator into cass-operator. This should occur via new struct images field in the CassDC CR.
4. The current mechanisms for setting images are based on string typed API fields which is not ideal or uniform even within k8ssandra-operator.
5. The existing string fields should be deprecated in favour of a structured image field.

[Miles-Garnsey]

It is worth noting that my commentary on expected behaviour above was based on comments to the effect that the string serverImage fields in the CassDC were deprecated in favour of the ImageConfig fields.

It appears that ImageConfig is not intended to replace serverImage at all, hence the confusion.

[Miles-Garnsey]

Also @burmanm and @jsanda, if you can both comment and tell me if you agree with my recommendations above, we can probably close this ticket in favour of tickets to implement them.

[burmanm]

No one has deprecated ServerImage. It even says in the description:

// This value will override anything set in the ImageConfig matching the ServerVersion

Both ConfigBuilderImage as well as SystemLoggerImage say the same. What I said is that these two are something that are hopefully going away completely and we should spend time doing that instead of making more configuration for something we wish to go away (and so deprecate them).