From 14128de0b534cfc001eda517182f527db1d72d4f Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Thu, 30 Mar 2023 14:07:29 +0200 Subject: [PATCH 01/11] Initial version --- .../pages/adr/ADR026-affinities.adoc | 2 +- .../pages/adr/ADR028-discovery-revision.adoc | 96 +++++++++++++++++++ 2 files changed, 97 insertions(+), 1 deletion(-) create mode 100644 modules/contributor/pages/adr/ADR028-discovery-revision.adoc diff --git a/modules/contributor/pages/adr/ADR026-affinities.adoc b/modules/contributor/pages/adr/ADR026-affinities.adoc index 8a2c97a6b..04ec4557a 100644 --- a/modules/contributor/pages/adr/ADR026-affinities.adoc +++ b/modules/contributor/pages/adr/ADR026-affinities.adoc @@ -1,7 +1,7 @@ = ADR026: Affinities Sebastian Bernauer v0.1, 2023-02-13 -:status: draft +:status: accepted * Status: {status} * Deciders: diff --git a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc new file mode 100644 index 000000000..b6784e5a5 --- /dev/null +++ b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc @@ -0,0 +1,96 @@ += ADR028: Discovery revision +Razvan Mihai +v0.1, 2023-03-30 +:status: draft + +* Status: {status} +* Contributors: +** Felix Hennig +** Malte Sander +** Natalie Klestrup Röijezon +** Razvan Mihai +** Sebastian Bernauer +* Date: 2023-02-28 + +== Context and Problem Statement + +// Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question. + +This ADR is written with a specific problems in mind, but the goal is to reach a generic mechanism for the discovery of products. +The current discovery mechanism is described https://docs.stackable.tech/home/stable/concepts/service_discovery.html[in our docs] (make sure to pick the 23.1 version). + +Basically when clients connect to products managed by Stackable, they need to have certain information about how to connect to these products. +Currently we expose some of these information, but not all (e.g. which ca cert the exposed product uses). + +== Decision Drivers +We have some common use-cases that we need to express via the discovery mechanism: + +1. Trino cluster +* Currently we don't write any discovery CM +* We need at least +** Coordinator address (currently only single coordinator is supported) +*** Ideally split it into the following attributes, so that clients can construct the URI. (trino python client e.g. needs all of these attributes separately and I don't want to rely on parsing and extracting the URI) +**** protocol (http/https) +**** host +**** port +** What Secretclass must be used to authenticate +*** null (no SecretClass): Means no authentication at all +*** (future) static: One of these plain credentials +*** tls: provides ca.crt that needs to have signed the client +*** ldap: +*** (future) kerberos: kdc where you can get a ticket from (together with the realm) +*** (future) oauth: +*** (future) jwt: + +2. HDFS cluster +* Currently we expose hdfs-site and core-site +* We need at least +** hdfs-site +** core-site +** What Secretclass must be used to authenticate +*** (future) kerberos: kdc where you can get a ticket from (together with the realm) +** The information about rpc encryption is already in the core-site, so not needed explicitly +** The information about data encryption is already in the hdfs-site, so not needed explicitly + +== Considered Options + +=== Option 1 + +==== Pros + + +==== Cons + +[source,yaml] +---- +status: + conditions: + - type: Available + status: "True" + lastProbeTime: 2023-02-28T14:02:00Z + lastTransitionTime: 2023-02-28T12:00:00Z + message: "UI and Postgres DB running" + - type: Degraded + status: "True" + lastProbeTime: 2023-02-28T14:02:00Z + lastTransitionTime: 2023-02-28T12:00:00Z + reason: "DruidConnection failed. " + - type: Progressing + status: "True" + lastProbeTime: 2023-02-28T14:02:00Z + lastTransitionTime: 2023-02-28T12:00:00Z + message: "New replicas starting." + - type: Upgradable + status: "Unknown" + lastProbeTime: 2023-02-28T14:02:00Z + lastTransitionTime: 2023-02-28T12:00:00Z + - type: Paused + status: "True" + lastProbeTime: 2023-02-28T14:02:00Z + lastTransitionTime: 2023-02-28T12:00:00Z + message: "User requested reconcile pause." +---- + +== Decision Outcome + +TODO From 5a1c80d8a2b64a08ab932cd8ac5889567c4ddcb0 Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Thu, 30 Mar 2023 15:38:58 +0200 Subject: [PATCH 02/11] Add some options --- .../pages/adr/ADR028-discovery-revision.adoc | 187 ++++++++++++++---- 1 file changed, 153 insertions(+), 34 deletions(-) diff --git a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc index b6784e5a5..dcd34a2d0 100644 --- a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc +++ b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc @@ -1,5 +1,5 @@ = ADR028: Discovery revision -Razvan Mihai +Sebastian Bernauer v0.1, 2023-03-30 :status: draft @@ -8,7 +8,6 @@ v0.1, 2023-03-30 ** Felix Hennig ** Malte Sander ** Natalie Klestrup Röijezon -** Razvan Mihai ** Sebastian Bernauer * Date: 2023-02-28 @@ -33,10 +32,11 @@ We have some common use-cases that we need to express via the discovery mechanis **** protocol (http/https) **** host **** port -** What Secretclass must be used to authenticate +*** In case of https: The SecretClass that provided the cert for the *server* +** What AuthenticationClass must be used to authenticate *** null (no SecretClass): Means no authentication at all *** (future) static: One of these plain credentials -*** tls: provides ca.crt that needs to have signed the client +*** tls: provides ca.crt that needs to have signed the *client* certificate *** ldap: *** (future) kerberos: kdc where you can get a ticket from (together with the realm) *** (future) oauth: @@ -47,50 +47,169 @@ We have some common use-cases that we need to express via the discovery mechanis * We need at least ** hdfs-site ** core-site -** What Secretclass must be used to authenticate +** What AuthenticationClass must be used to authenticate *** (future) kerberos: kdc where you can get a ticket from (together with the realm) -** The information about rpc encryption is already in the core-site, so not needed explicitly -** The information about data encryption is already in the hdfs-site, so not needed explicitly +** The information about rpc encryption is already in the core-site, so need to expose it explicitly +** The information about data encryption is already in the hdfs-site, so need to expose it explicitly == Considered Options -=== Option 1 +=== TLS: Discovery config contains SecretClass +The discovery includes the SecretClass used to obtain the *server* certificate + +Trino discovery: +[source,yaml] +---- +metadata: + name: my-trino +coordinatorEndpoint: + host: trino-coordinator.ns.svc.cluster.local + port: 8443 + protocol: + http: {} + # OR + https: + caCertSecretClass: tls +--- +# superset config +security: + tls: + secretClassName: tls + kerberos: + secretClassName: kerberos +backends: # Don't look at the Superset CRD structure, we are only interested in the tls stuff here + - name: my-trino + trino: + discoveryConfigName: my-trino +---- + +==== Pros + +==== Cons + +=== TLS: Client needs to specify SecretClass +--- +The discovery does *not* include the SecretClass used to obtain the *server* certificate. +Instead the client must specify which SecretClass should be used to verify the *server* certificate. +For usability reasons it can be omitted and defaults to the SecretClass the client uses for itself. + +Trino discovery: +[source,yaml] +---- +metadata: + name: my-trino +coordinatorEndpoint: https://trino-coordinator.ns.svc.cluster.local:8443 +--- +# superset config +security: + tls: + secretClassName: tls + kerberos: + secretClassName: kerberos +backends: # Don't look at the Superset CRD structure, we are only interested in the tls stuff here + - name: my-trino + trino: + discoveryConfigName: my-trino + # override tls from the global config, OPTIONALLY + tlsSecretClass: my-second-pki +---- + +==== Pros + +* Operator does not need to read/look at the DiscoveryConfig (as we can statically set up the secret-op tls secretClass volumes rather than retrieving them from the DiscoveryConfig). +* Some clients only support a single pki, in that case we could not give the ability to overwrite the secretClass coming from the product itself. + +==== Cons + +* The client has to know what pki/secretClass the server is using. + +=== TLS: Include caCert in Discovery config + +Trino discovery: +[source,yaml] +---- +metadata: + name: my-trino +endpoint: + host: trino-coordinator.ns.svc.cluster.local + port: 8443 + protocol: + http: {} + # OR + https: + caBundle: | # Containing a PEM certificate + === BEGIN CERTIFICATE === + XXX + === END CERTIFICATE === +---- ==== Pros +* Assuming DiscoveryConfig is located within a CM, the operator can simply mount the discovery CM to get the ca.crt. Operator does not need to read/look at the DiscoveryConfig. +* Easier for external clients to use as they don't need to know the concept of SecretClasses and don't even need to run withing k8s. +* The client has to *not* know what pki/secretClass the server is using. ==== Cons +* BIG QUESTION: How should the product operator get the ca cert from the SecretClass it uses to get the *server* cert from? +** The secret-op could e.g. offer an HTTP api to fetch the ca.crt of a given SecretClass or e.g. write the ca.crt into the status of a SecretClass + + +=== Authentication: Add AuthenticationClass to Discovery Config + +Trino discovery: +[source,yaml] +---- +metadata: + name: my-trino +authentication: + authenticationClass: my-class +---- + +==== Cons + +* Operator has to read the AuthenticationClass to determine its type (pw/tls/keytab) and set up the needed volumes and commands. +* The AuthenticationClass is meant to describe "how should a server verify connecting clients" and re-purpose it to mean "how a client should authenticate itself". + + +=== Authentication: Add SecretClass to Discovery Config + +Trino discovery: [source,yaml] ---- -status: - conditions: - - type: Available - status: "True" - lastProbeTime: 2023-02-28T14:02:00Z - lastTransitionTime: 2023-02-28T12:00:00Z - message: "UI and Postgres DB running" - - type: Degraded - status: "True" - lastProbeTime: 2023-02-28T14:02:00Z - lastTransitionTime: 2023-02-28T12:00:00Z - reason: "DruidConnection failed. " - - type: Progressing - status: "True" - lastProbeTime: 2023-02-28T14:02:00Z - lastTransitionTime: 2023-02-28T12:00:00Z - message: "New replicas starting." - - type: Upgradable - status: "Unknown" - lastProbeTime: 2023-02-28T14:02:00Z - lastTransitionTime: 2023-02-28T12:00:00Z - - type: Paused - status: "True" - lastProbeTime: 2023-02-28T14:02:00Z - lastTransitionTime: 2023-02-28T12:00:00Z - message: "User requested reconcile pause." +metadata: + name: my-trino +authentication: + secretClass: client-tls # Use this SecretClass to obtain your credentials (regardless of type of SecretClass) ---- +==== Cons + +* Operator has to read the SecretClass to determine its type (pw/tls/keytab) and set up the needed volumes and commands. + + +=== Authentication: Add needed details + +Trino discovery: +[source,yaml] +---- +metadata: + name: my-trino +authentication: + none: {} + password: {} + tls: + secretClass: client-tls # Use this SecretClass to obtain a *client* cert + kerberos: + secretClass: client-tls # Use this SecretClass to obtain a keytab + oauth: + secretClass: client-tls # Use this SecretClass to obtain whatever it needs +---- + +==== Pros + +* Operator has *not* to read the SecretClass to determine its type (pw/tls/keytab), as the type is already encoded in the Discovery config. + == Decision Outcome TODO From a4d160c5288b005db219aaaf9c11477fa8f65b48 Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Mon, 3 Apr 2023 13:44:50 +0200 Subject: [PATCH 03/11] WIP --- .../pages/adr/ADR028-discovery-revision.adoc | 148 ++++++++++++++++-- 1 file changed, 139 insertions(+), 9 deletions(-) diff --git a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc index dcd34a2d0..0219269a9 100644 --- a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc +++ b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc @@ -54,7 +54,110 @@ We have some common use-cases that we need to express via the discovery mechanis == Considered Options -=== TLS: Discovery config contains SecretClass +=== [1] Discovery Object: Use ConfigMap + +Use a ConfigMap: + +[source,yaml] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + name: simple-hdfs +spec: + data: + hdfs-site.xml: + core-site.xml: + ca.crt: | # Containing a PEM certificate + === BEGIN CERTIFICATE === + XXX + === END CERTIFICATE === + authentication: | + kerberos: + secretClass: client-tls # Use this SecretClass to obtain a keytab + # Alternative solution without yaml enum: + authenticationType: Kerberos + authenticationSecretClass: client-kerberos +---- + +==== Pros + +* Easier to use for consuming applications outside of Stackable, as they can simply mount the CM or download it to a file. + On the other hand when we start to put in yaml objects that need to be parsed we would loose the benefit. + +==== Cons + +* No complex structure such as enums (we can mitigate this by sticking in custom yaml into the CM). + Users don't have any form of validation when creating their own discovery CM e.g. pointing to their existing HDFS. + +=== [1] Discovery Object: Use dedicated CRD object for every product + +Or use a dedicated HdfsClusterDiscovery crd: + +[source,yaml] +---- +apiVersion: hdfs.stackable.tech/v1alpha1 +kind: HdfsClusterDiscovery +metadata: + name: simple-hdfs +spec: + hdfs-site.xml: # xml + core-site.xml: # xml + httpProtocol: + http: {} + # OR + https: + caBundle: | # Containing a PEM certificate + === BEGIN CERTIFICATE === + XXX + === END CERTIFICATE === + authentication: | + kerberos: + secretClass: client-tls # Use this SecretClass to obtain a keytab +---- + +==== Pros + +==== Cons + +* Operator A needs to compile against operator b to have access to it's discovery struct. An alternative would be to put the Discovery CRDs in operator-rs. +* Operator versioning hell. On the other hand we have the same problem with ConfigMaps, as e.g. a newly introduced key is missing because of an older hdfs operator version. + +=== [1] Discovery Object: Use dedicated CRD object shared between all products + +Or use a dedicated ClusterDiscovery crd: + +[source,yaml] +---- +apiVersion: discovery.stackable.tech/v1alpha1 +kind: ClusterDiscovery +metadata: + name: simple-hdfs +spec: + # Whatever +---- + +==== Pros + +* Only one struct in operator-rs -> No cross-operator dependencies. + +==== Cons + +* It does not seem like it's possible to find a common struct all products can agree upon + +=== [1] Discovery Object: Write the discovery to Product CR status + +Instead of writing discovery information to dedicated objects - such as CM or custom CR - we "simply" write the discovery information to the status of the Cluster CR. + +==== Pros + +==== Cons + +* It does not enable users to bring their own product and talk to it from Stackable, e.g. a user-provided HDFS. +* It does not allow things such as a ZNode for Zookeeper as we either use the Zookeeper CR for discovery or we use a ZNode but than can't use a Zookeeper CR. + Currently we have the freedom of either connection to a Zookeeper root dir or a ZNode transparently. + +=== [2] TLS: Discovery config contains SecretClass The discovery includes the SecretClass used to obtain the *server* certificate Trino discovery: @@ -87,7 +190,7 @@ backends: # Don't look at the Superset CRD structure, we are only interested in ==== Cons -=== TLS: Client needs to specify SecretClass +=== [2] TLS: Client needs to specify SecretClass --- The discovery does *not* include the SecretClass used to obtain the *server* certificate. Instead the client must specify which SecretClass should be used to verify the *server* certificate. @@ -123,7 +226,7 @@ backends: # Don't look at the Superset CRD structure, we are only interested in * The client has to know what pki/secretClass the server is using. -=== TLS: Include caCert in Discovery config +=== [2] TLS: Include caCert in Discovery config Trino discovery: [source,yaml] @@ -155,7 +258,7 @@ endpoint: ** The secret-op could e.g. offer an HTTP api to fetch the ca.crt of a given SecretClass or e.g. write the ca.crt into the status of a SecretClass -=== Authentication: Add AuthenticationClass to Discovery Config +=== [3] Authentication: Add AuthenticationClass to Discovery Config Trino discovery: [source,yaml] @@ -172,7 +275,7 @@ authentication: * The AuthenticationClass is meant to describe "how should a server verify connecting clients" and re-purpose it to mean "how a client should authenticate itself". -=== Authentication: Add SecretClass to Discovery Config +=== [3] Authentication: Add SecretClass to Discovery Config Trino discovery: [source,yaml] @@ -188,7 +291,7 @@ authentication: * Operator has to read the SecretClass to determine its type (pw/tls/keytab) and set up the needed volumes and commands. -=== Authentication: Add needed details +=== [3] Authentication: Add needed details Trino discovery: [source,yaml] @@ -201,15 +304,42 @@ authentication: tls: secretClass: client-tls # Use this SecretClass to obtain a *client* cert kerberos: - secretClass: client-tls # Use this SecretClass to obtain a keytab + secretClass: client-kerberos # Use this SecretClass to obtain a keytab oauth: - secretClass: client-tls # Use this SecretClass to obtain whatever it needs + secretClass: client-oauth # Use this SecretClass to obtain whatever it needs ---- ==== Pros * Operator has *not* to read the SecretClass to determine its type (pw/tls/keytab), as the type is already encoded in the Discovery config. +==== Cons + +* Operator has read the Discovery CM it wants to connect to + == Decision Outcome -TODO +[1] Discovery Object: TODO +[2] TLS: TODO +[3] Authentication: TODO + +=== Appendix A +Let's model a kerberos secured HDFS with the Options "TLS: Include caCert in Discovery config" and "Authentication: Add needed details" + +[source,yaml] +---- +apiVersion: hdfs.stackable.tech/v1alpha1 +kind: HdfsCluster +metadata: + name: simple-hdfs +spec: + zookeeperConfigMapName: simple-hdfs-znode + nameNodes: {} + dataNodes: {} + journalNodes: {} + # TODO Refine CRD + kerberos: + tlsSecretClass: tls + kerberosSecretClass: kerberos + wireEncryption: Privacy +---- From d3b6fdd429b66def37820143b4ddc1d9ee7b6bc3 Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Wed, 23 Aug 2023 12:55:44 +0200 Subject: [PATCH 04/11] WIP --- .../pages/adr/ADR028-discovery-revision.adoc | 90 ++++++++++++++++--- 1 file changed, 80 insertions(+), 10 deletions(-) diff --git a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc index 0219269a9..81c045bff 100644 --- a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc +++ b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc @@ -15,11 +15,13 @@ v0.1, 2023-03-30 // Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question. -This ADR is written with a specific problems in mind, but the goal is to reach a generic mechanism for the discovery of products. +This ADR is written with specific problems in mind, but the goal is to reach a generic mechanism for the discovery of products. The current discovery mechanism is described https://docs.stackable.tech/home/stable/concepts/service_discovery.html[in our docs] (make sure to pick the 23.1 version). Basically when clients connect to products managed by Stackable, they need to have certain information about how to connect to these products. Currently we expose some of these information, but not all (e.g. which ca cert the exposed product uses). +On the other hand it could be the case that users have external services which Stackable services should use, e.g. +an non-stackable HDFS where an Stackable Trino should connect to. == Decision Drivers We have some common use-cases that we need to express via the discovery mechanism: @@ -35,11 +37,11 @@ We have some common use-cases that we need to express via the discovery mechanis *** In case of https: The SecretClass that provided the cert for the *server* ** What AuthenticationClass must be used to authenticate *** null (no SecretClass): Means no authentication at all -*** (future) static: One of these plain credentials +*** static: One of these plain credentials *** tls: provides ca.crt that needs to have signed the *client* certificate *** ldap: *** (future) kerberos: kdc where you can get a ticket from (together with the realm) -*** (future) oauth: +*** (future) oauth/oidc: *** (future) jwt: 2. HDFS cluster @@ -48,9 +50,9 @@ We have some common use-cases that we need to express via the discovery mechanis ** hdfs-site ** core-site ** What AuthenticationClass must be used to authenticate -*** (future) kerberos: kdc where you can get a ticket from (together with the realm) -** The information about rpc encryption is already in the core-site, so need to expose it explicitly -** The information about data encryption is already in the hdfs-site, so need to expose it explicitly +*** kerberos: kdc where you can get a ticket from (together with the realm) +** The information about rpc encryption is already in the core-site, so no need to expose it explicitly +** The information about data encryption is already in the hdfs-site, so no need to expose it explicitly == Considered Options @@ -84,11 +86,13 @@ spec: * Easier to use for consuming applications outside of Stackable, as they can simply mount the CM or download it to a file. On the other hand when we start to put in yaml objects that need to be parsed we would loose the benefit. +* Single API call to retrieve all running products ==== Cons * No complex structure such as enums (we can mitigate this by sticking in custom yaml into the CM). Users don't have any form of validation when creating their own discovery CM e.g. pointing to their existing HDFS. +* Cannot have two products with the same name, as the discovery CM name clashes. One solution could be to prefix the product name (e.g. trino-simple), This can impose other problems such as too long CM names. === [1] Discovery Object: Use dedicated CRD object for every product @@ -111,17 +115,65 @@ spec: === BEGIN CERTIFICATE === XXX === END CERTIFICATE === - authentication: | + authentication: kerberos: secretClass: client-tls # Use this SecretClass to obtain a keytab ---- ==== Pros +* Validation by using e.g. complex enums +* Commons structure can be shared between all operators, such as `Listener` endpoints or tls server certificate information + ==== Cons * Operator A needs to compile against operator b to have access to it's discovery struct. An alternative would be to put the Discovery CRDs in operator-rs. * Operator versioning hell. On the other hand we have the same problem with ConfigMaps, as e.g. a newly introduced key is missing because of an older hdfs operator version. +* Dependant Pods (such as hbase on hdfs) can not simply mount a CM containing the hdfs-site and core-site. Instead the hbase-operator needs to read the HdfsClusterDiscovery, copy the hdfs-site and core-site into a CM and than mount that into the hbase Pods. This can be solved by the HdfsClusterDiscovery to point to a CM that contains hdfs-site and core-site xmls. +* Multiple API calls need to retrieve all running Stackable service (in stackablectl or cockpit). This would be a single API call in case of discovery CM or a shared CRD for all product discoveries. +* Side-Note: `stackablectl stacklet list` should *not* look at discovery objects, as they can come from a user and are external systems, where we don't know anything about. + +=== [1] Discovery Object: Use dedicated CRD object for every product - in combination with ConfigMap + +Or use a dedicated HdfsClusterDiscovery crd: + +[source,yaml] +---- +# This struct should *not* contain any information than any client possible wants to mount +# Instead put these kind of information into the CM +# +# This struct resides in a new repo stackable-discovery and is pulled in as a dependency in (possibly) operator-rs and all operators. +apiVersion: hdfs.stackable.tech/v1alpha1 +kind: HdfsClusterDiscovery +metadata: + name: simple-hdfs +spec: + hdfsSitesConfigMap: hdfs-simple-hdfs + productVersion: 3.3.4 + httpProtocol: + http: {} + # OR + https: + caSecretClass: tls + authentication: + kerberos: + keytabSecretClass: client-tls # Use this SecretClass to obtain a keytab +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: hdfs-simple-hdfs # prefix to avoid naming collisions +spec: + data: + hdfs-site.xml: + core-site.xml: +---- + +==== Pros + +* Fixes mount problem from `Discovery Object: Use dedicated CRD object for every product` + +==== Cons === [1] Discovery Object: Use dedicated CRD object shared between all products @@ -134,16 +186,34 @@ kind: ClusterDiscovery metadata: name: simple-hdfs spec: - # Whatever + productVersion: 3.3.4 + hdfs: # same structure as in HdfsClusterDiscovery example + hdfsSitesConfigMap: hdfs-simple-hdfs + httpProtocol: + http: {} + # OR + https: + caSecretClass: tls + authentication: + kerberos: + keytabSecretClass: client-tls # Use this SecretClass to obtain a keytab + # OR + hbase: # Whatever + # OR + zookeeper: # Whatever + # ... ---- ==== Pros -* Only one struct in operator-rs -> No cross-operator dependencies. +* Only one struct in operator-rs => No cross-operator dependencies. +* Single API call to retrieve all stackable products. Question is if this really helps a lot, as callers probably also are interested in the status of the product, which needs further API calls (irrelevant - see Cons). ==== Cons -* It does not seem like it's possible to find a common struct all products can agree upon +* All product discoveries are versioned together. E.g. a new mandatory field for hdfs requires all operators to bump the Discovery CRD to `v2`. We hope that this does not happen too often. +* Names can collide +* `stackablectl stacklet list` should *not* look at discovery objects, as they can come from a user and are external systems, where we don't know anything about. So in case we want to introduce a `Stacklet` object listing anyway, so the `Pro` regarding the API calls is irrelevant. === [1] Discovery Object: Write the discovery to Product CR status From f2158cf38e5d56f66ac58d33f08160a66152ebe7 Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Wed, 23 Aug 2023 16:31:13 +0200 Subject: [PATCH 05/11] WIP --- .../pages/adr/ADR028-discovery-revision.adoc | 101 ++++++++++++------ 1 file changed, 69 insertions(+), 32 deletions(-) diff --git a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc index 81c045bff..8916bd151 100644 --- a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc +++ b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc @@ -23,6 +23,10 @@ Currently we expose some of these information, but not all (e.g. which ca cert t On the other hand it could be the case that users have external services which Stackable services should use, e.g. an non-stackable HDFS where an Stackable Trino should connect to. +Question 1: How do we want to make products discoverable? +Question 2: How should clients verify the identity of the server (e.g. the cert of the server) +Question 3: How should clients authenticate themselves against the server? + == Decision Drivers We have some common use-cases that we need to express via the discovery mechanism: @@ -148,8 +152,8 @@ kind: HdfsClusterDiscovery metadata: name: simple-hdfs spec: + productVersion: 3.3.4 # *could* be put in common struct and #[serde(flattened)] hdfsSitesConfigMap: hdfs-simple-hdfs - productVersion: 3.3.4 httpProtocol: http: {} # OR @@ -169,6 +173,26 @@ spec: core-site.xml: ---- +[source,yaml] +---- +apiVersion: trino.stackable.tech/v1alpha1 +kind: TrinoClusterDiscovery +metadata: + name: simple-trino +spec: + productVersion: "414" + coordinatorEndpoint: + host: trino-coordinator.ns.svc.cluster.local + port: 8443 + protocol: + http: {} + # OR + https: + caCertSecretClass: tls + authentication: , see below +# No CM needed +---- + ==== Pros * Fixes mount problem from `Discovery Object: Use dedicated CRD object for every product` @@ -228,32 +252,33 @@ Instead of writing discovery information to dedicated objects - such as CM or cu Currently we have the freedom of either connection to a Zookeeper root dir or a ZNode transparently. === [2] TLS: Discovery config contains SecretClass -The discovery includes the SecretClass used to obtain the *server* certificate +The discovery includes the SecretClass used to obtain the ca.crt used to validate the *server* certificate Trino discovery: [source,yaml] ---- +apiVersion: trino.stackable.tech/v1alpha1 +kind: TrinoClusterDiscovery metadata: - name: my-trino -coordinatorEndpoint: - host: trino-coordinator.ns.svc.cluster.local - port: 8443 - protocol: - http: {} - # OR - https: - caCertSecretClass: tls + name: simple-trino +spec: + coordinatorEndpoint: + host: trino-coordinator.ns.svc.cluster.local + port: 8443 + protocol: + http: {} + # OR + https: + caCertSecretClass: tls # gives ca.crt used to verify the server cert --- # superset config security: - tls: + tls: # server tls cert secretClassName: tls - kerberos: - secretClassName: kerberos backends: # Don't look at the Superset CRD structure, we are only interested in the tls stuff here - name: my-trino trino: - discoveryConfigName: my-trino + discovery: my-trino ---- ==== Pros @@ -269,20 +294,26 @@ For usability reasons it can be omitted and defaults to the SecretClass the clie Trino discovery: [source,yaml] ---- +apiVersion: trino.stackable.tech/v1alpha1 +kind: TrinoClusterDiscovery metadata: - name: my-trino -coordinatorEndpoint: https://trino-coordinator.ns.svc.cluster.local:8443 ---- + name: simple-trino +spec: + coordinatorEndpoint: + host: trino-coordinator.ns.svc.cluster.local + port: 8443 + protocol: + http: {} + # OR + https: {} # NO! cert information # superset config security: tls: secretClassName: tls - kerberos: - secretClassName: kerberos backends: # Don't look at the Superset CRD structure, we are only interested in the tls stuff here - name: my-trino trino: - discoveryConfigName: my-trino + discovery: my-trino # override tls from the global config, OPTIONALLY tlsSecretClass: my-second-pki ---- @@ -295,6 +326,7 @@ backends: # Don't look at the Superset CRD structure, we are only interested in ==== Cons * The client has to know what pki/secretClass the server is using. +* Superset TrinoConnection could not only say "Connect this Superset and this Trino", but need to say "using this ca.crt to validate the Trino server" === [2] TLS: Include caCert in Discovery config @@ -318,7 +350,6 @@ endpoint: ==== Pros -* Assuming DiscoveryConfig is located within a CM, the operator can simply mount the discovery CM to get the ca.crt. Operator does not need to read/look at the DiscoveryConfig. * Easier for external clients to use as they don't need to know the concept of SecretClasses and don't even need to run withing k8s. * The client has to *not* know what pki/secretClass the server is using. @@ -339,11 +370,13 @@ authentication: authenticationClass: my-class ---- +==== Pros +* IMPORTANT: This is the only thing the server can know (how he is verifying client identities). He can not recommend an SecretClass used to obtain the client credentials. E.g. he uses an LDAP AuthenticationClass, there is no way it can now what SecretClass provides credentials accepted by LDAP. (Most cases it will be a user logging into a WebUI and the LDAP credentials of the user are not even stored anywhere but just remembered by the user) + ==== Cons * Operator has to read the AuthenticationClass to determine its type (pw/tls/keytab) and set up the needed volumes and commands. -* The AuthenticationClass is meant to describe "how should a server verify connecting clients" and re-purpose it to mean "how a client should authenticate itself". - +// * The AuthenticationClass is meant to describe "how should a server verify connecting clients" and re-purpose it to mean "how a client should authenticate itself". Image a user creates a Secret `trino-users` with *only* a ca.crt and a SecretClass `trino-users` on top. The connecting client than has no way of knowing how to get a client cert. === [3] Authentication: Add SecretClass to Discovery Config @@ -359,7 +392,7 @@ authentication: ==== Cons * Operator has to read the SecretClass to determine its type (pw/tls/keytab) and set up the needed volumes and commands. - +* Image then SecretClass is of type `k8sSearch`. The connection client (e.g. controlled via superset-operator) than has no idea if he should expect a tls.crd or a keytab when mounting the SecretClass. === [3] Authentication: Add needed details @@ -372,11 +405,9 @@ authentication: none: {} password: {} tls: - secretClass: client-tls # Use this SecretClass to obtain a *client* cert + secretClass: client-tls # Use this SecretClass to obtain a *client* cert tls.crt kerberos: secretClass: client-kerberos # Use this SecretClass to obtain a keytab - oauth: - secretClass: client-oauth # Use this SecretClass to obtain whatever it needs ---- ==== Pros @@ -385,13 +416,19 @@ authentication: ==== Cons -* Operator has read the Discovery CM it wants to connect to +=== [3] Authentication: Don't add information how to authenticate + +Trino discovery does not provide any information on how to authenticate + +==== Cons + +* Not viable, as users need to know how to connect, and are not expected to try 50 different auth methods. We need to give them a AuthenticationClass, that says them e.g. what LDAP or PKI is used. == Decision Outcome -[1] Discovery Object: TODO -[2] TLS: TODO -[3] Authentication: TODO +[1] Discovery Object: `Discovery Object: Use dedicated CRD object for every product - in combination with ConfigMap` +[2] Server tls cert: TODO +[3] Authentication: `Authentication: Add AuthenticationClass to Discovery Config` === Appendix A Let's model a kerberos secured HDFS with the Options "TLS: Include caCert in Discovery config" and "Authentication: Add needed details" From f9dbbfad49a3876b05e4f7818cb648328b9b913a Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Thu, 24 Aug 2023 10:50:41 +0200 Subject: [PATCH 06/11] WIP --- .../pages/adr/ADR028-discovery-revision.adoc | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc index 8916bd151..d5df9ec85 100644 --- a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc +++ b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc @@ -359,6 +359,45 @@ endpoint: ** The secret-op could e.g. offer an HTTP api to fetch the ca.crt of a given SecretClass or e.g. write the ca.crt into the status of a SecretClass +=== [2] TLS: Include SecretClass in discovery, user can override it + +Trino discovery: +[source,yaml] +---- +apiVersion: trino.stackable.tech/v1alpha1 +kind: TrinoClusterDiscovery +metadata: + name: simple-trino +spec: + coordinatorEndpoint: + host: trino-coordinator.ns.svc.cluster.local + port: 8443 + protocol: + http: {} + # OR + https: + caCertSecretClass: tls # gives ca.crt used to verify the server cert +--- +# superset config +security: + tls: # server tls cert + secretClassName: tls +backends: # Don't look at the Superset CRD structure, we are only interested in the tls stuff here + - name: my-trino + trino: + discovery: my-trino + # OPTIONALLY override the spec.coordinatorEndpoint.protocol.https.caCertSecretClass coming from TrinoClusterDiscovery + tlsSecretClass: my-second-pki +---- + +==== Pros + +* Compromise with all usability and flexibility + +==== Cons + +* Less secure by default + === [3] Authentication: Add AuthenticationClass to Discovery Config Trino discovery: From 806a4a8e3c402924a7797ef40a8def9f0ec5e987 Mon Sep 17 00:00:00 2001 From: Techassi Date: Fri, 25 Aug 2023 14:02:27 +0200 Subject: [PATCH 07/11] Add contributors --- modules/contributor/pages/adr/ADR028-discovery-revision.adoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc index d5df9ec85..c71aba7ac 100644 --- a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc +++ b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc @@ -9,6 +9,9 @@ v0.1, 2023-03-30 ** Malte Sander ** Natalie Klestrup Röijezon ** Sebastian Bernauer +** Andrew Kenworthy +** Lukas Voetmand +** Sascha Lautenschläger * Date: 2023-02-28 == Context and Problem Statement From 70b9f12f5848d84873d817b64dde21ba2cea9293 Mon Sep 17 00:00:00 2001 From: Techassi Date: Fri, 25 Aug 2023 14:02:38 +0200 Subject: [PATCH 08/11] Adjust formatting --- .../pages/adr/ADR028-discovery-revision.adoc | 128 ++++++++++++------ 1 file changed, 84 insertions(+), 44 deletions(-) diff --git a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc index c71aba7ac..bbb94ac3a 100644 --- a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc +++ b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc @@ -16,28 +16,33 @@ v0.1, 2023-03-30 == Context and Problem Statement -// Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question. +// Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to +// articulate the problem in form of a question. -This ADR is written with specific problems in mind, but the goal is to reach a generic mechanism for the discovery of products. -The current discovery mechanism is described https://docs.stackable.tech/home/stable/concepts/service_discovery.html[in our docs] (make sure to pick the 23.1 version). +This ADR is written with specific problems in mind, but the goal is to reach a generic mechanism for the discovery of +products. The current discovery mechanism is described +https://docs.stackable.tech/home/stable/concepts/service_discovery.html[in our docs] (make sure to pick the 23.1 +version). -Basically when clients connect to products managed by Stackable, they need to have certain information about how to connect to these products. -Currently we expose some of these information, but not all (e.g. which ca cert the exposed product uses). -On the other hand it could be the case that users have external services which Stackable services should use, e.g. -an non-stackable HDFS where an Stackable Trino should connect to. +Basically when clients connect to products managed by Stackable, they need to have certain information about how to +connect to these products. Currently we expose some of these information, but not all (e.g. which ca cert the exposed +product uses). On the other hand it could be the case that users have external services which Stackable services should +use, e.g. an non-stackable HDFS where an Stackable Trino should connect to. -Question 1: How do we want to make products discoverable? -Question 2: How should clients verify the identity of the server (e.g. the cert of the server) -Question 3: How should clients authenticate themselves against the server? +*Question 1:* How do we want to make products discoverable? +*Question 2:* How should clients verify the identity of the server (e.g. the cert of the server) +*Question 3:* How should clients authenticate themselves against the server? == Decision Drivers + We have some common use-cases that we need to express via the discovery mechanism: 1. Trino cluster * Currently we don't write any discovery CM * We need at least ** Coordinator address (currently only single coordinator is supported) -*** Ideally split it into the following attributes, so that clients can construct the URI. (trino python client e.g. needs all of these attributes separately and I don't want to rely on parsing and extracting the URI) +*** Ideally split it into the following attributes, so that clients can construct the URI. (trino python client e.g. + needs all of these attributes separately and I don't want to rely on parsing and extracting the URI) **** protocol (http/https) **** host **** port @@ -91,15 +96,16 @@ spec: ==== Pros -* Easier to use for consuming applications outside of Stackable, as they can simply mount the CM or download it to a file. - On the other hand when we start to put in yaml objects that need to be parsed we would loose the benefit. +* Easier to use for consuming applications outside of Stackable, as they can simply mount the CM or download it to a + file. On the other hand when we start to put in yaml objects that need to be parsed we would loose the benefit. * Single API call to retrieve all running products ==== Cons -* No complex structure such as enums (we can mitigate this by sticking in custom yaml into the CM). - Users don't have any form of validation when creating their own discovery CM e.g. pointing to their existing HDFS. -* Cannot have two products with the same name, as the discovery CM name clashes. One solution could be to prefix the product name (e.g. trino-simple), This can impose other problems such as too long CM names. +* No complex structure such as enums (we can mitigate this by sticking in custom yaml into the CM). Users don't have any + form of validation when creating their own discovery CM e.g. pointing to their existing HDFS. +* Cannot have two products with the same name, as the discovery CM name clashes. One solution could be to prefix the + product name (e.g. trino-simple), This can impose other problems such as too long CM names. === [1] Discovery Object: Use dedicated CRD object for every product @@ -130,15 +136,23 @@ spec: ==== Pros * Validation by using e.g. complex enums -* Commons structure can be shared between all operators, such as `Listener` endpoints or tls server certificate information +* Commons structure can be shared between all operators, such as `Listener` endpoints or tls server certificate + information ==== Cons -* Operator A needs to compile against operator b to have access to it's discovery struct. An alternative would be to put the Discovery CRDs in operator-rs. -* Operator versioning hell. On the other hand we have the same problem with ConfigMaps, as e.g. a newly introduced key is missing because of an older hdfs operator version. -* Dependant Pods (such as hbase on hdfs) can not simply mount a CM containing the hdfs-site and core-site. Instead the hbase-operator needs to read the HdfsClusterDiscovery, copy the hdfs-site and core-site into a CM and than mount that into the hbase Pods. This can be solved by the HdfsClusterDiscovery to point to a CM that contains hdfs-site and core-site xmls. -* Multiple API calls need to retrieve all running Stackable service (in stackablectl or cockpit). This would be a single API call in case of discovery CM or a shared CRD for all product discoveries. -* Side-Note: `stackablectl stacklet list` should *not* look at discovery objects, as they can come from a user and are external systems, where we don't know anything about. +* Operator A needs to compile against operator b to have access to it's discovery struct. An alternative would be to put + the Discovery CRDs in operator-rs. +* Operator versioning hell. On the other hand we have the same problem with ConfigMaps, as e.g. a newly introduced key + is missing because of an older hdfs operator version. +* Dependant Pods (such as hbase on hdfs) can not simply mount a CM containing the hdfs-site and core-site. Instead the + hbase-operator needs to read the HdfsClusterDiscovery, copy the hdfs-site and core-site into a CM and than mount that + into the hbase Pods. This can be solved by the HdfsClusterDiscovery to point to a CM that contains hdfs-site and + core-site xmls. +* Multiple API calls need to retrieve all running Stackable service (in stackablectl or cockpit). This would be a single + API call in case of discovery CM or a shared CRD for all product discoveries. +* Side-Note: `stackablectl stacklet list` should *not* look at discovery objects, as they can come from a user and are + external systems, where we don't know anything about. === [1] Discovery Object: Use dedicated CRD object for every product - in combination with ConfigMap @@ -234,27 +248,34 @@ spec: ==== Pros * Only one struct in operator-rs => No cross-operator dependencies. -* Single API call to retrieve all stackable products. Question is if this really helps a lot, as callers probably also are interested in the status of the product, which needs further API calls (irrelevant - see Cons). +* Single API call to retrieve all stackable products. Question is if this really helps a lot, as callers probably also + are interested in the status of the product, which needs further API calls (irrelevant - see Cons). ==== Cons -* All product discoveries are versioned together. E.g. a new mandatory field for hdfs requires all operators to bump the Discovery CRD to `v2`. We hope that this does not happen too often. +* All product discoveries are versioned together. E.g. a new mandatory field for hdfs requires all operators to bump the + Discovery CRD to `v2`. We hope that this does not happen too often. * Names can collide -* `stackablectl stacklet list` should *not* look at discovery objects, as they can come from a user and are external systems, where we don't know anything about. So in case we want to introduce a `Stacklet` object listing anyway, so the `Pro` regarding the API calls is irrelevant. +* `stackablectl stacklet list` should *not* look at discovery objects, as they can come from a user and are external + systems, where we don't know anything about. So in case we want to introduce a `Stacklet` object listing anyway, so + the `Pro` regarding the API calls is irrelevant. === [1] Discovery Object: Write the discovery to Product CR status -Instead of writing discovery information to dedicated objects - such as CM or custom CR - we "simply" write the discovery information to the status of the Cluster CR. +Instead of writing discovery information to dedicated objects - such as CM or custom CR - we "simply" write the +discovery information to the status of the Cluster CR. ==== Pros ==== Cons * It does not enable users to bring their own product and talk to it from Stackable, e.g. a user-provided HDFS. -* It does not allow things such as a ZNode for Zookeeper as we either use the Zookeeper CR for discovery or we use a ZNode but than can't use a Zookeeper CR. - Currently we have the freedom of either connection to a Zookeeper root dir or a ZNode transparently. +* It does not allow things such as a ZNode for Zookeeper as we either use the Zookeeper CR for discovery or we use a + ZNode but than can't use a Zookeeper CR. Currently we have the freedom of either connection to a Zookeeper root dir or + a ZNode transparently. === [2] TLS: Discovery config contains SecretClass + The discovery includes the SecretClass used to obtain the ca.crt used to validate the *server* certificate Trino discovery: @@ -290,9 +311,10 @@ backends: # Don't look at the Superset CRD structure, we are only interested in === [2] TLS: Client needs to specify SecretClass --- -The discovery does *not* include the SecretClass used to obtain the *server* certificate. -Instead the client must specify which SecretClass should be used to verify the *server* certificate. -For usability reasons it can be omitted and defaults to the SecretClass the client uses for itself. + +The discovery does *not* include the SecretClass used to obtain the *server* certificate. Instead the client must +specify which SecretClass should be used to verify the *server* certificate. For usability reasons it can be omitted and +defaults to the SecretClass the client uses for itself. Trino discovery: [source,yaml] @@ -323,13 +345,16 @@ backends: # Don't look at the Superset CRD structure, we are only interested in ==== Pros -* Operator does not need to read/look at the DiscoveryConfig (as we can statically set up the secret-op tls secretClass volumes rather than retrieving them from the DiscoveryConfig). -* Some clients only support a single pki, in that case we could not give the ability to overwrite the secretClass coming from the product itself. +* Operator does not need to read/look at the DiscoveryConfig (as we can statically set up the secret-op tls secretClass + volumes rather than retrieving them from the DiscoveryConfig). +* Some clients only support a single pki, in that case we could not give the ability to overwrite the secretClass coming + from the product itself. ==== Cons * The client has to know what pki/secretClass the server is using. -* Superset TrinoConnection could not only say "Connect this Superset and this Trino", but need to say "using this ca.crt to validate the Trino server" +* Superset TrinoConnection could not only say "Connect this Superset and this Trino", but need to say "using this ca.crt + to validate the Trino server" === [2] TLS: Include caCert in Discovery config @@ -353,18 +378,22 @@ endpoint: ==== Pros -* Easier for external clients to use as they don't need to know the concept of SecretClasses and don't even need to run withing k8s. +* Easier for external clients to use as they don't need to know the concept of SecretClasses and don't even need to run + withing k8s. * The client has to *not* know what pki/secretClass the server is using. ==== Cons -* BIG QUESTION: How should the product operator get the ca cert from the SecretClass it uses to get the *server* cert from? -** The secret-op could e.g. offer an HTTP api to fetch the ca.crt of a given SecretClass or e.g. write the ca.crt into the status of a SecretClass +* BIG QUESTION: How should the product operator get the ca cert from the SecretClass it uses to get the *server* cert + from? +** The secret-op could e.g. offer an HTTP api to fetch the ca.crt of a given SecretClass or e.g. write the ca.crt into + the status of a SecretClass === [2] TLS: Include SecretClass in discovery, user can override it Trino discovery: + [source,yaml] ---- apiVersion: trino.stackable.tech/v1alpha1 @@ -413,12 +442,18 @@ authentication: ---- ==== Pros -* IMPORTANT: This is the only thing the server can know (how he is verifying client identities). He can not recommend an SecretClass used to obtain the client credentials. E.g. he uses an LDAP AuthenticationClass, there is no way it can now what SecretClass provides credentials accepted by LDAP. (Most cases it will be a user logging into a WebUI and the LDAP credentials of the user are not even stored anywhere but just remembered by the user) +* IMPORTANT: This is the only thing the server can know (how he is verifying client identities). He can not recommend an + SecretClass used to obtain the client credentials. E.g. he uses an LDAP AuthenticationClass, there is no way it can + now what SecretClass provides credentials accepted by LDAP. (Most cases it will be a user logging into a WebUI and the + LDAP credentials of the user are not even stored anywhere but just remembered by the user) ==== Cons -* Operator has to read the AuthenticationClass to determine its type (pw/tls/keytab) and set up the needed volumes and commands. -// * The AuthenticationClass is meant to describe "how should a server verify connecting clients" and re-purpose it to mean "how a client should authenticate itself". Image a user creates a Secret `trino-users` with *only* a ca.crt and a SecretClass `trino-users` on top. The connecting client than has no way of knowing how to get a client cert. +* Operator has to read the AuthenticationClass to determine its type (pw/tls/keytab) and set up the needed volumes and + commands. +// * The AuthenticationClass is meant to describe "how should a server verify connecting clients" and re-purpose it to +// mean "how a client should authenticate itself". Image a user creates a Secret `trino-users` with *only* a ca.crt +// and a SecretClass `trino-users` on top. The connecting client than has no way of knowing how to get a client cert. === [3] Authentication: Add SecretClass to Discovery Config @@ -434,7 +469,8 @@ authentication: ==== Cons * Operator has to read the SecretClass to determine its type (pw/tls/keytab) and set up the needed volumes and commands. -* Image then SecretClass is of type `k8sSearch`. The connection client (e.g. controlled via superset-operator) than has no idea if he should expect a tls.crd or a keytab when mounting the SecretClass. +* Image then SecretClass is of type `k8sSearch`. The connection client (e.g. controlled via superset-operator) than has + no idea if he should expect a tls.crd or a keytab when mounting the SecretClass. === [3] Authentication: Add needed details @@ -454,7 +490,8 @@ authentication: ==== Pros -* Operator has *not* to read the SecretClass to determine its type (pw/tls/keytab), as the type is already encoded in the Discovery config. +* Operator has *not* to read the SecretClass to determine its type (pw/tls/keytab), as the type is already encoded in + the Discovery config. ==== Cons @@ -464,7 +501,8 @@ Trino discovery does not provide any information on how to authenticate ==== Cons -* Not viable, as users need to know how to connect, and are not expected to try 50 different auth methods. We need to give them a AuthenticationClass, that says them e.g. what LDAP or PKI is used. +* Not viable, as users need to know how to connect, and are not expected to try 50 different auth methods. We need to + give them a AuthenticationClass, that says them e.g. what LDAP or PKI is used. == Decision Outcome @@ -473,7 +511,9 @@ Trino discovery does not provide any information on how to authenticate [3] Authentication: `Authentication: Add AuthenticationClass to Discovery Config` === Appendix A -Let's model a kerberos secured HDFS with the Options "TLS: Include caCert in Discovery config" and "Authentication: Add needed details" + +Let's model a kerberos secured HDFS with the Options "TLS: Include caCert in Discovery config" and "Authentication: +Add needed details" [source,yaml] ---- From b4bab29be94a04eef2fee7fd168706a96129de97 Mon Sep 17 00:00:00 2001 From: Techassi Date: Fri, 25 Aug 2023 15:12:52 +0200 Subject: [PATCH 09/11] More formatting --- .../pages/adr/ADR028-discovery-revision.adoc | 137 ++++++++++-------- 1 file changed, 77 insertions(+), 60 deletions(-) diff --git a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc index bbb94ac3a..e5f52f874 100644 --- a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc +++ b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc @@ -1,4 +1,4 @@ -= ADR028: Discovery revision += ADR028: Service Discovery (Revision) Sebastian Bernauer v0.1, 2023-03-30 :status: draft @@ -68,9 +68,15 @@ We have some common use-cases that we need to express via the discovery mechanis == Considered Options -=== [1] Discovery Object: Use ConfigMap +=== Discovery Object -Use a ConfigMap: +This Discovery Object is written by the Product Operator. It describes how the product can be accessed by other +connecting entities (clients). These clients can be inside the same Kubernetes cluster, in an external Kubernetes +cluster or deployed using a different approach like running inside a VM or on bare metal hardware. The discovery object +is used both internally by the SDP and by other actors (for example users). Each Discovery Object is created in the same +namespace the product runs in. This means cross-namespace discovery needs special attention. + +==== Discovery Object: Option 1 - ConfigMap [source,yaml] ---- @@ -94,22 +100,20 @@ spec: authenticationSecretClass: client-kerberos ---- -==== Pros +===== Pros * Easier to use for consuming applications outside of Stackable, as they can simply mount the CM or download it to a file. On the other hand when we start to put in yaml objects that need to be parsed we would loose the benefit. * Single API call to retrieve all running products -==== Cons +===== Cons * No complex structure such as enums (we can mitigate this by sticking in custom yaml into the CM). Users don't have any form of validation when creating their own discovery CM e.g. pointing to their existing HDFS. * Cannot have two products with the same name, as the discovery CM name clashes. One solution could be to prefix the product name (e.g. trino-simple), This can impose other problems such as too long CM names. -=== [1] Discovery Object: Use dedicated CRD object for every product - -Or use a dedicated HdfsClusterDiscovery crd: +==== Discovery Object: Option 2 - Dedicated CRD Object for every Product [source,yaml] ---- @@ -133,30 +137,28 @@ spec: secretClass: client-tls # Use this SecretClass to obtain a keytab ---- -==== Pros +===== Pros * Validation by using e.g. complex enums * Commons structure can be shared between all operators, such as `Listener` endpoints or tls server certificate information -==== Cons +===== Cons -* Operator A needs to compile against operator b to have access to it's discovery struct. An alternative would be to put +* Operator A needs to compile against operator B to have access to it's discovery struct. An alternative would be to put the Discovery CRDs in operator-rs. * Operator versioning hell. On the other hand we have the same problem with ConfigMaps, as e.g. a newly introduced key is missing because of an older hdfs operator version. * Dependant Pods (such as hbase on hdfs) can not simply mount a CM containing the hdfs-site and core-site. Instead the hbase-operator needs to read the HdfsClusterDiscovery, copy the hdfs-site and core-site into a CM and than mount that into the hbase Pods. This can be solved by the HdfsClusterDiscovery to point to a CM that contains hdfs-site and - core-site xmls. + core-site XML files. * Multiple API calls need to retrieve all running Stackable service (in stackablectl or cockpit). This would be a single API call in case of discovery CM or a shared CRD for all product discoveries. * Side-Note: `stackablectl stacklet list` should *not* look at discovery objects, as they can come from a user and are external systems, where we don't know anything about. -=== [1] Discovery Object: Use dedicated CRD object for every product - in combination with ConfigMap - -Or use a dedicated HdfsClusterDiscovery crd: +==== Discovery Object: Option 3 - Dedicated CRD Object + ConfigMap for every Product [source,yaml] ---- @@ -210,15 +212,13 @@ spec: # No CM needed ---- -==== Pros +===== Pros * Fixes mount problem from `Discovery Object: Use dedicated CRD object for every product` -==== Cons - -=== [1] Discovery Object: Use dedicated CRD object shared between all products +===== Cons -Or use a dedicated ClusterDiscovery crd: +==== Discovery Object: Option 4 - Dedicated CRD Object shared between all Products [source,yaml] ---- @@ -245,13 +245,13 @@ spec: # ... ---- -==== Pros +===== Pros * Only one struct in operator-rs => No cross-operator dependencies. * Single API call to retrieve all stackable products. Question is if this really helps a lot, as callers probably also are interested in the status of the product, which needs further API calls (irrelevant - see Cons). -==== Cons +===== Cons * All product discoveries are versioned together. E.g. a new mandatory field for hdfs requires all operators to bump the Discovery CRD to `v2`. We hope that this does not happen too often. @@ -260,25 +260,30 @@ spec: systems, where we don't know anything about. So in case we want to introduce a `Stacklet` object listing anyway, so the `Pro` regarding the API calls is irrelevant. -=== [1] Discovery Object: Write the discovery to Product CR status +==== Discovery Object: Option 5 - Write Discovery to Product CR Status Instead of writing discovery information to dedicated objects - such as CM or custom CR - we "simply" write the discovery information to the status of the Cluster CR. -==== Pros +===== Pros -==== Cons +* None currently + +===== Cons * It does not enable users to bring their own product and talk to it from Stackable, e.g. a user-provided HDFS. * It does not allow things such as a ZNode for Zookeeper as we either use the Zookeeper CR for discovery or we use a ZNode but than can't use a Zookeeper CR. Currently we have the freedom of either connection to a Zookeeper root dir or a ZNode transparently. -=== [2] TLS: Discovery config contains SecretClass +''' -The discovery includes the SecretClass used to obtain the ca.crt used to validate the *server* certificate +=== TLS + +==== TLS: Option 1 - Discovery Config contains SecretClass + +The discovery includes the SecretClass used to obtain the ca.crt used to validate the *server* certificate. -Trino discovery: [source,yaml] ---- apiVersion: trino.stackable.tech/v1alpha1 @@ -305,18 +310,20 @@ backends: # Don't look at the Superset CRD structure, we are only interested in discovery: my-trino ---- -==== Pros +===== Pros -==== Cons +* Currently none -=== [2] TLS: Client needs to specify SecretClass ---- +===== Cons + +* Currently none + +==== TLS: Option 2 - Client specifies SecretClass The discovery does *not* include the SecretClass used to obtain the *server* certificate. Instead the client must specify which SecretClass should be used to verify the *server* certificate. For usability reasons it can be omitted and defaults to the SecretClass the client uses for itself. -Trino discovery: [source,yaml] ---- apiVersion: trino.stackable.tech/v1alpha1 @@ -343,22 +350,21 @@ backends: # Don't look at the Superset CRD structure, we are only interested in tlsSecretClass: my-second-pki ---- -==== Pros +===== Pros * Operator does not need to read/look at the DiscoveryConfig (as we can statically set up the secret-op tls secretClass volumes rather than retrieving them from the DiscoveryConfig). * Some clients only support a single pki, in that case we could not give the ability to overwrite the secretClass coming from the product itself. -==== Cons +===== Cons * The client has to know what pki/secretClass the server is using. * Superset TrinoConnection could not only say "Connect this Superset and this Trino", but need to say "using this ca.crt to validate the Trino server" -=== [2] TLS: Include caCert in Discovery config +==== TLS: Option 3 - Include caCert in Discovery Config -Trino discovery: [source,yaml] ---- metadata: @@ -376,23 +382,21 @@ endpoint: === END CERTIFICATE === ---- -==== Pros +===== Pros * Easier for external clients to use as they don't need to know the concept of SecretClasses and don't even need to run withing k8s. * The client has to *not* know what pki/secretClass the server is using. -==== Cons +===== Cons -* BIG QUESTION: How should the product operator get the ca cert from the SecretClass it uses to get the *server* cert +* BIG QUESTION: How should the product operator get the CA cert from the SecretClass it uses to get the *server* cert from? ** The secret-op could e.g. offer an HTTP api to fetch the ca.crt of a given SecretClass or e.g. write the ca.crt into the status of a SecretClass -=== [2] TLS: Include SecretClass in discovery, user can override it - -Trino discovery: +==== TLS: Option 4 - Include SecretClass in Discovery (User can override it) [source,yaml] ---- @@ -422,17 +426,20 @@ backends: # Don't look at the Superset CRD structure, we are only interested in tlsSecretClass: my-second-pki ---- -==== Pros +===== Pros * Compromise with all usability and flexibility -==== Cons +===== Cons * Less secure by default -=== [3] Authentication: Add AuthenticationClass to Discovery Config +''' + +=== Authentication + +==== Authentication: Option 1 - Add AuthenticationClass to Discovery Config -Trino discovery: [source,yaml] ---- metadata: @@ -441,13 +448,14 @@ authentication: authenticationClass: my-class ---- -==== Pros +===== Pros + * IMPORTANT: This is the only thing the server can know (how he is verifying client identities). He can not recommend an SecretClass used to obtain the client credentials. E.g. he uses an LDAP AuthenticationClass, there is no way it can now what SecretClass provides credentials accepted by LDAP. (Most cases it will be a user logging into a WebUI and the LDAP credentials of the user are not even stored anywhere but just remembered by the user) -==== Cons +===== Cons * Operator has to read the AuthenticationClass to determine its type (pw/tls/keytab) and set up the needed volumes and commands. @@ -455,9 +463,8 @@ authentication: // mean "how a client should authenticate itself". Image a user creates a Secret `trino-users` with *only* a ca.crt // and a SecretClass `trino-users` on top. The connecting client than has no way of knowing how to get a client cert. -=== [3] Authentication: Add SecretClass to Discovery Config +==== Authentication: Option 2 - Add SecretClass to Discovery Config -Trino discovery: [source,yaml] ---- metadata: @@ -466,13 +473,17 @@ authentication: secretClass: client-tls # Use this SecretClass to obtain your credentials (regardless of type of SecretClass) ---- -==== Cons +===== Pros + +* Currently none + +===== Cons * Operator has to read the SecretClass to determine its type (pw/tls/keytab) and set up the needed volumes and commands. * Image then SecretClass is of type `k8sSearch`. The connection client (e.g. controlled via superset-operator) than has no idea if he should expect a tls.crd or a keytab when mounting the SecretClass. -=== [3] Authentication: Add needed details +==== Authentication: Option 3 - Add needed Details Trino discovery: [source,yaml] @@ -488,27 +499,33 @@ authentication: secretClass: client-kerberos # Use this SecretClass to obtain a keytab ---- -==== Pros +===== Pros * Operator has *not* to read the SecretClass to determine its type (pw/tls/keytab), as the type is already encoded in the Discovery config. -==== Cons +===== Cons -=== [3] Authentication: Don't add information how to authenticate +* Currently none + +==== Authentication: Option 4 - Provide no Authentication Information Trino discovery does not provide any information on how to authenticate -==== Cons +===== Pros + +* Currently none + +===== Cons * Not viable, as users need to know how to connect, and are not expected to try 50 different auth methods. We need to give them a AuthenticationClass, that says them e.g. what LDAP or PKI is used. == Decision Outcome -[1] Discovery Object: `Discovery Object: Use dedicated CRD object for every product - in combination with ConfigMap` -[2] Server tls cert: TODO -[3] Authentication: `Authentication: Add AuthenticationClass to Discovery Config` +* *Discovery Object:* Option 3 +* *Server TLS:* TBD, but leaning towards Option 2 +* *Authentication:* Option 1 === Appendix A From 22737cbb4e7feef4f46be945e1927b37ba481f3b Mon Sep 17 00:00:00 2001 From: Techassi Date: Fri, 25 Aug 2023 15:41:18 +0200 Subject: [PATCH 10/11] More polish --- .../pages/adr/ADR028-discovery-revision.adoc | 64 ++++++++++--------- .../contributor/partials/current_adrs.adoc | 1 + 2 files changed, 34 insertions(+), 31 deletions(-) diff --git a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc index e5f52f874..f47b85656 100644 --- a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc +++ b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc @@ -29,9 +29,9 @@ connect to these products. Currently we expose some of these information, but no product uses). On the other hand it could be the case that users have external services which Stackable services should use, e.g. an non-stackable HDFS where an Stackable Trino should connect to. -*Question 1:* How do we want to make products discoverable? -*Question 2:* How should clients verify the identity of the server (e.g. the cert of the server) -*Question 3:* How should clients authenticate themselves against the server? +* *Question 1:* How do we want to make products discoverable? +* *Question 2:* How should clients verify the identity of the server (e.g. the cert of the server) +* *Question 3:* How should clients authenticate themselves against the server? == Decision Drivers @@ -100,13 +100,13 @@ spec: authenticationSecretClass: client-kerberos ---- -===== Pros +===== *Pros* * Easier to use for consuming applications outside of Stackable, as they can simply mount the CM or download it to a file. On the other hand when we start to put in yaml objects that need to be parsed we would loose the benefit. * Single API call to retrieve all running products -===== Cons +===== *Cons* * No complex structure such as enums (we can mitigate this by sticking in custom yaml into the CM). Users don't have any form of validation when creating their own discovery CM e.g. pointing to their existing HDFS. @@ -137,13 +137,13 @@ spec: secretClass: client-tls # Use this SecretClass to obtain a keytab ---- -===== Pros +===== *Pros* * Validation by using e.g. complex enums * Commons structure can be shared between all operators, such as `Listener` endpoints or tls server certificate information -===== Cons +===== *Cons* * Operator A needs to compile against operator B to have access to it's discovery struct. An alternative would be to put the Discovery CRDs in operator-rs. @@ -212,11 +212,13 @@ spec: # No CM needed ---- -===== Pros +===== *Pros* * Fixes mount problem from `Discovery Object: Use dedicated CRD object for every product` -===== Cons +===== *Cons* + +* Currently none ==== Discovery Object: Option 4 - Dedicated CRD Object shared between all Products @@ -245,13 +247,13 @@ spec: # ... ---- -===== Pros +===== *Pros* * Only one struct in operator-rs => No cross-operator dependencies. * Single API call to retrieve all stackable products. Question is if this really helps a lot, as callers probably also are interested in the status of the product, which needs further API calls (irrelevant - see Cons). -===== Cons +===== *Cons* * All product discoveries are versioned together. E.g. a new mandatory field for hdfs requires all operators to bump the Discovery CRD to `v2`. We hope that this does not happen too often. @@ -265,11 +267,11 @@ spec: Instead of writing discovery information to dedicated objects - such as CM or custom CR - we "simply" write the discovery information to the status of the Cluster CR. -===== Pros +===== *Pros* * None currently -===== Cons +===== *Cons* * It does not enable users to bring their own product and talk to it from Stackable, e.g. a user-provided HDFS. * It does not allow things such as a ZNode for Zookeeper as we either use the Zookeeper CR for discovery or we use a @@ -310,11 +312,11 @@ backends: # Don't look at the Superset CRD structure, we are only interested in discovery: my-trino ---- -===== Pros +===== *Pros* * Currently none -===== Cons +===== *Cons* * Currently none @@ -350,14 +352,14 @@ backends: # Don't look at the Superset CRD structure, we are only interested in tlsSecretClass: my-second-pki ---- -===== Pros +===== *Pros* * Operator does not need to read/look at the DiscoveryConfig (as we can statically set up the secret-op tls secretClass volumes rather than retrieving them from the DiscoveryConfig). * Some clients only support a single pki, in that case we could not give the ability to overwrite the secretClass coming from the product itself. -===== Cons +===== *Cons* * The client has to know what pki/secretClass the server is using. * Superset TrinoConnection could not only say "Connect this Superset and this Trino", but need to say "using this ca.crt @@ -382,13 +384,13 @@ endpoint: === END CERTIFICATE === ---- -===== Pros +===== *Pros* * Easier for external clients to use as they don't need to know the concept of SecretClasses and don't even need to run withing k8s. * The client has to *not* know what pki/secretClass the server is using. -===== Cons +===== *Cons* * BIG QUESTION: How should the product operator get the CA cert from the SecretClass it uses to get the *server* cert from? @@ -426,11 +428,11 @@ backends: # Don't look at the Superset CRD structure, we are only interested in tlsSecretClass: my-second-pki ---- -===== Pros +===== *Pros* * Compromise with all usability and flexibility -===== Cons +===== *Cons* * Less secure by default @@ -448,14 +450,14 @@ authentication: authenticationClass: my-class ---- -===== Pros +===== *Pros* -* IMPORTANT: This is the only thing the server can know (how he is verifying client identities). He can not recommend an - SecretClass used to obtain the client credentials. E.g. he uses an LDAP AuthenticationClass, there is no way it can +* *IMPORTANT:* This is the only thing the server can know (how he is verifying client identities). He can not recommend + an SecretClass used to obtain the client credentials. E.g. he uses an LDAP AuthenticationClass, there is no way it can now what SecretClass provides credentials accepted by LDAP. (Most cases it will be a user logging into a WebUI and the LDAP credentials of the user are not even stored anywhere but just remembered by the user) -===== Cons +===== *Cons* * Operator has to read the AuthenticationClass to determine its type (pw/tls/keytab) and set up the needed volumes and commands. @@ -473,11 +475,11 @@ authentication: secretClass: client-tls # Use this SecretClass to obtain your credentials (regardless of type of SecretClass) ---- -===== Pros +===== *Pros* * Currently none -===== Cons +===== *Cons* * Operator has to read the SecretClass to determine its type (pw/tls/keytab) and set up the needed volumes and commands. * Image then SecretClass is of type `k8sSearch`. The connection client (e.g. controlled via superset-operator) than has @@ -499,12 +501,12 @@ authentication: secretClass: client-kerberos # Use this SecretClass to obtain a keytab ---- -===== Pros +===== *Pros* * Operator has *not* to read the SecretClass to determine its type (pw/tls/keytab), as the type is already encoded in the Discovery config. -===== Cons +===== *Cons* * Currently none @@ -512,11 +514,11 @@ authentication: Trino discovery does not provide any information on how to authenticate -===== Pros +===== *Pros* * Currently none -===== Cons +===== *Cons* * Not viable, as users need to know how to connect, and are not expected to try 50 different auth methods. We need to give them a AuthenticationClass, that says them e.g. what LDAP or PKI is used. diff --git a/modules/contributor/partials/current_adrs.adoc b/modules/contributor/partials/current_adrs.adoc index 1415b3127..780a9ef40 100644 --- a/modules/contributor/partials/current_adrs.adoc +++ b/modules/contributor/partials/current_adrs.adoc @@ -24,3 +24,4 @@ **** xref:adr/ADR025-logging_architecture.adoc[] **** xref:adr/ADR026-affinities.adoc[] **** xref:adr/ADR027-status.adoc[] +**** xref:adr/ADR028-discovery-revision.adoc[] From 00450c4291482531188b8b2c642ff6edd0cba3a3 Mon Sep 17 00:00:00 2001 From: Techassi Date: Fri, 25 Aug 2023 17:00:24 +0200 Subject: [PATCH 11/11] Start adding content for discovery operator --- .../pages/adr/ADR028-discovery-revision.adoc | 66 ++++++++++++++++++- 1 file changed, 64 insertions(+), 2 deletions(-) diff --git a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc index f47b85656..e6821afc6 100644 --- a/modules/contributor/pages/adr/ADR028-discovery-revision.adoc +++ b/modules/contributor/pages/adr/ADR028-discovery-revision.adoc @@ -112,6 +112,7 @@ spec: form of validation when creating their own discovery CM e.g. pointing to their existing HDFS. * Cannot have two products with the same name, as the discovery CM name clashes. One solution could be to prefix the product name (e.g. trino-simple), This can impose other problems such as too long CM names. +* Cannot be mounted across namespaces, would need to be copied. ==== Discovery Object: Option 2 - Dedicated CRD Object for every Product @@ -141,7 +142,8 @@ spec: * Validation by using e.g. complex enums * Commons structure can be shared between all operators, such as `Listener` endpoints or tls server certificate - information + information. One possible repository for such code would be either in `operator-rs` directly, as a separate workspace + member or in a completely separate repo. One possible name could be `stackable_discovery`. ===== *Cons* @@ -158,6 +160,9 @@ spec: * Side-Note: `stackablectl stacklet list` should *not* look at discovery objects, as they can come from a user and are external systems, where we don't know anything about. +Many of above cons could be solved/mitigated by the introduction of a Discovery controller/operator. See +xref:#discovery-op[here] how this operator/controller would work. + ==== Discovery Object: Option 3 - Dedicated CRD Object + ConfigMap for every Product [source,yaml] @@ -523,13 +528,70 @@ Trino discovery does not provide any information on how to authenticate * Not viable, as users need to know how to connect, and are not expected to try 50 different auth methods. We need to give them a AuthenticationClass, that says them e.g. what LDAP or PKI is used. +[#discovery-op] +=== Discovery Operator/Controller + +[source] +---- + NS 1 || NS 2 + ----------- || ------------ + | HDFS-Op | || --- | Trino-Op | <-------- + ----------- || | ------------ | + | || | | | | + deploys || | deploys | + | || | | | | + v || | v v | +------------------------ || | ------ ---------- | +| HDFSClusterDiscovery | || | | CM | | STS(s) | | +| | || | ------ ---------- | +| - hdfs-site.xml | || | | +| - core-site.xml | || | | +| - auth: ... | || wants to | +------------------------ || connect and uses + ^ || creates | + | || | | + | || v | + | || ------------ -------------- | + watches || | Request* | | Response** | -- + | || ------------ -------------- + | || ^ ^ + | || | | + | || watches creates + | || | | + | ---------------- | | + -------- | Discovery-Op | ---------------- + ---------------- + || +---- + +The Discovery Operator/Controller handles cross-namespace discovery between products. Each product operator creates a +namespace-scoped Discovery object (See `HDFSClusterDiscovery`). However some Discovery objects may need additional +ConfigMaps. These ConfigMaps cannot be mounted across namespaces, which would prevent products from being deployed into +separate namespaces. To solve this issue, the Discovery Operator would handle cross-namespace discovery. An example +discovery flow would look like this: + +. The Trino-Op (the Product/Client to be exact) located in `NS 2` wants to connect to the HDFS cluster in `NS 1` +. The Trino-Op then creates a `Request*` object for which the Discovery-Op watches +. When the Discovery-Op encounters a new `Request*`, it will lookup the requested endpoint +. Gathered information from the endpoints Discovery data will by forwarded by the Discovery-Op by creating a + `Response**` in the target namespace `NS 2` +. The Trino-Op then uses the `Response**` to fill out local ConfigMaps. The change in a ConfigMap/STS triggers an + automatic restart of the product. This immediately propagates changes in the endpoints Discovery. + +pass:[*] Exact type to be determined. + +pass:[**] Exact type to be determined, re-using the Discovery type might be a solution, but will entangle types across +operators. One possible solution might by to create one repository named `stackable_discovery`, which includes the +Discovery-Op and all request and response types. Product operators then only need to depend on one additional crate +`stackable_discovery` to enable them to use the request and response types. + == Decision Outcome * *Discovery Object:* Option 3 * *Server TLS:* TBD, but leaning towards Option 2 * *Authentication:* Option 1 -=== Appendix A +== Appendix A Let's model a kerberos secured HDFS with the Options "TLS: Include caCert in Discovery config" and "Authentication: Add needed details"