Where do we find documentation about how to best spec manifest about mTLS on Cassandra/DSE on Kubernetes, cert-manager integration and gotchas?

[mparikhcloudbeds]

Cass-Operator adopters who have encryption and authentication over mTLS requirements from the organization, would want to look for detailed documentation, best practices and gotchas when applying Cassandra/DSE deployment automation via Cass-Operator.

Some organizations would have also adopted kubernetes cert-manager for mTLS capabilities. It would be really important to have a guideline around a path forward of this integration with cass-operator.

The guideline may also need expansion on multi-DC (single or multi-region) setup with do's and don'ts.
Snippets of this implementation in practice and any case studies would aid for sure.

This request can be broken into multiple tickets and areas of concerns as the team see the fit.

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: CASS-53

[bradfordcp]

@johnsmartco let's sync up on the code paths here, required figures, and configuration excerpts. This is a very interesting topic for a number of users.

[jdonenine]

@johnsmartco and @bradfordcp my thinking here is that we'll dump some of the background we've already started to collect here in this ticket -- that's why I asked @mparikhcloudbeds to create it, and then we can use this is as the basis for the enhancements to the official docs -- this and probably also the blog post that @Miles-Garnsey is preparing to publish on the subject.

@jsanda can you dump the background on how cass-operator works w.r.t. encryption here?

[jsanda]

Sure, I will try to provide a brief summary what cass-operator does.

If only for my own edification I want to first provide some background.

Background
For internode encryption we are talking about the following properties in cassandra.yaml:

server_encryption_options:
  internode_encryption: all
  keystore: keystore.p12
  keystore_password: cassandra
  truststore: truststore.p12
  truststore_password: cassandra
  require_client_auth: true

If we want mTLS we need require_client_auth: true.

While the format of the keystore and truststore is the same, they serve different purposes. The keystore is used to provide credentials for verification by a client. A truststore is used to authenticate a client. In a Cassandra cluster (set aside Kubernetes for the moment), all of the nodes might be configured to use a common truststore that has a CA cert. Each one should ideally be configure with its own keystore that contains a node-specific private key and cert.

In Kubernetes setting up per-node keystores is non-trivial since StatefulSets do not allow pod-specific configurations. I am not sure how to do what I described in the preceding paragraph with Statefulsets. It is nonetheless good background.

Lastly, the same file can be used as both the keystore and the truststore. It is better to not do this, but it does work.

Internode encryption in cass-operator
Now let's go through what cass-operator does during its reconciliation loop:

• Fetch the CA cert secret
  • cass-operator looks for a secret having a name with the format <dc-name>-ca-keystore
• If the secret does not exist, create the secret, CA cert, and private key
  • Create CA cert and private key
    • The cert is configured to expire in a year
    • The cert's common name is set to <dc-name>-ca-keystore.<namespace>.svc
  • Create CA secret that contains the cert and private key. Persist it.
  • Configure keystore
    • Generate a PKCS8 formatted private key
    • Create a signed certificate
    • Create a keystore that includes the certificate, private key, and CA cert
  • Create keystore secret named <dc-name>-keystore
• Create a volume named encryption-cred-storage that is mounted at /etc/encryption

It is possible to configure the CassandraDatacenter with additional volumes for mounted secrets. Today this has to be done with the podTemplateSpec property. I can provide an example of that in a follow up comment. We do this in k8ssandra :)

[jsanda]

Here is an example manifest that uses custom keystore and truststores:

apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: dc1
spec:
  size: 1
  serverType: cassandra
  clusterName: dev
  config:
    cassandra-yaml:
      server_encryption_options:
        internode_encryption: all
        require_client_auth: true
        require_endpoint_verification: true
        keystore_type: JKS
        keystore: /mnt/keystore/keystore.jks
        keystore_password: cassandra
        truststore_type: JKS
        truststore: /mnt/truststore/truststore.jks
        truststore_password: cassandra
    jvm-server-options:
      initial_heap_size: "800M"
      max_heap_size: "800M"
  serverVersion: 4.0.1
  dockerImageRunsAsCassandra: true
  storageConfig:
    cassandraDataVolumeClaimSpec:
      storageClassName: standard
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
  podTemplateSpec:
    containers:
      - name: cassandra
        volumeMounts:
          - name: keystore-secret
            mountPath: /mnt/keystore
          - name: truststore-secret
            mountPath: /mnt/truststore
    volumes:
      - name: keystore-secret
        secret:
          secretName: keystore
      - name: trustore-secret
        secret:
          secretName: truststore

The example assumes that the keystore and truststore secrets already exist.

The cassandra container declaration:

    containers:
      - name: cassandra
        volumeMounts:
          - name: keystore-secret
            mountPath: /mnt/keystore
          - name: truststore-secret
            mountPath: /mnt/truststore

is a partial declaration. The volumeMounts will be merged with the default ones provided by cas-operator. Similarly the volumes will be merged with the ones provided by cass-operator.

[mparikhcloudbeds]

@jsanda - How does the makeup of truststore look like? What are the necessary files in there?

[jsanda]

In its simplest form the truststore should contain the public key of the CA cert that is used to sign the C* node certificates. I recommend the following articles for background:

• https://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html
• https://thelastpickle.com/blog/2019/07/18/hostname-verification.html

[jsanda]

@mparikhcloudbeds is there anything else you need for this issue? If not, can we close it?