Operator v1: Support multiple external listeners in Cluster CRD

[paulzhang97]

We support only one external listener in each of the API endpoints (Kafka, Proxy, and Schema Registry) in Cluster CRD. This PR is to support multiple external listeners in Cluster CRD.

[chrisseto]

Is there a reason to not use json/yaml.Marshal here? If the need is to avoid quoting of potentially templated values, you should be able to work around that with a custom type and omitempty:

type TemplatedString string

type TemplatedInt string

func (t TemplatedString) MarshalYAML() {
// format the value so it's not escaped here. It should be possible to generate invalid JSON/YAML though I've not tried it myself...
}

[paulzhang97]

You make me think it again 😄 . The issue here is how to wrap key and value with single quote '. I still could not figure out how to do it.
We need to encode it like { 'name': 'listener-name', 'address':'0.0.0.0'} rather than { "name": "listener-name", "address": "0.0.0.0" }.

[chrisseto]

Why do we need to encode them with single quotes?

EDIT:

If that's the constraint, feel we'd be better off regexing the quotes after JSON marshaling than maintaining all this custom serialization code. Something like: s/(^|[^\\])"/'/g should do it. You just need to be sure to not replace escaped quotes in strings which is what the capture group there should do. Though a negative lookbehind should work as well if you're familiar with them and go's regex engine supports them. I'd like to better understand they why of single quotes before going down that path though ;)

[paulzhang97]

I haven't looked into why single quotes are needed. I feel that like it has something to do with config.Set().

I like your proposal. I will give it a try.

[paulzhang97]

Well. I made hybrid changes. See the last commit.

I failed to write MarshalJSON for allListenersTemplateSpec, since a list type of field inside the struct needs to be encoded as e.g. "redpanda.kafka_api":"[{...},{...}]" (with double quotes on the list value, the value needs to be a string) instead of redpanda.kafka_api":[{...},{...}]. I tried to pass the one without double quotes to Configurator, it does not work. Did not dig into much. Bottom of line, I would not want to change how it works today since it is how Cloudv2 configures additional listeners for Private Link today.

Any other idea?

[chrisseto]

config.Set just calls yaml.Unmarshal (With some... interesting preprocessing). I see no reason why single quotes would be required.

// Set sets a field in pointer-to-struct p to a value, following yaml tags.
//
//	Key:    string containing the yaml field tags, e.g: 'rpk.admin_api'.
//	Value:  string representation of the value
func Set[T any](p *T, key, value string) error {
	if key == "" {
		return fmt.Errorf("key field must not be empty")
	}
	tags := strings.Split(key, ".")
	for _, tag := range tags {
		if _, _, err := splitTagIndex(tag); err != nil {
			return err
		}
	}
	finalTag := tags[len(tags)-1]
	if len(tags) > 1 && (finalTag == "enabled" && tags[len(tags)-2] == "tls" || finalTag == "tls") {
		switch value {
		case "{}":
		case "null":
		case "true":
			value = "{}"
		case "false":
			value = "null"
		default:
			// If the final tag is 'tls', it might be a value. So we continue
			// and handle below.
			if finalTag != "tls" {
				return fmt.Errorf("%s must be true or {}", key)
			}
		}
		if finalTag == "enabled" {
			tags = tags[:len(tags)-1]
			finalTag = tags[len(tags)-1]
		}
	}

	field, other, err := getField(tags, "", reflect.ValueOf(p).Elem())
	if err != nil {
		return err
	}
	isOther := other != reflect.Value{}

	// For Other fields, we need to wrap the value in key:value format when
	// unmarshaling, and we forbid indexing.
	if isOther {
		if _, index, _ := splitTagIndex(finalTag); index >= 0 {
			return fmt.Errorf("cannot index into unknown field %q", finalTag)
		}
		field = other
	}

	if !field.CanAddr() {
		return errors.New("rpk bug, please describe how you encountered this at https://github.com/redpanda-data/redpanda/issues/new?assignees=&labels=kind%2Fbug&template=01_bug_report.md")
	}

	if isOther {
		value = fmt.Sprintf("%s: %s", finalTag, value)
	}

	// If we cannot unmarshal, but our error looks like we are trying to
	// unmarshal a single element into a slice, we index[0] into the slice
	// and try unmarshaling again.
	rawv := []byte(value)
	if err := yaml.Unmarshal(rawv, field.Addr().Interface()); err != nil {
		// First we try wrapped with [ and ].
		if wrapped, ok := tryValueAsUnwrappedArray(field, value, err); ok {
			if err := yaml.Unmarshal([]byte(wrapped), field.Addr().Interface()); err == nil {
				return nil
			}
		}
		// If that still fails, we try setting a slice value if the
		// target is a slice.
		if elem0, ok := tryValueAsSlice0(field, err); ok {
			return yaml.Unmarshal(rawv, elem0.Addr().Interface())
		}
		return err
	}
	return nil
}

It also seems like it's be reasonably easy to side step config.Set in favor of directly setting these values in the configurator. Something like:

res, err := utils.Compute(v, utils.NewEndpointTemplateData(hostIndex, hostIP, hostIndexOffset), false)
if err != nil {
	return err
}

var addlListeners []config.NamedAuthNSocketAddress
yaml.Unmarshal(*&res, addListeners)

nodeConfig.Redpanda.KafkaAPI = append(nodeConfig.Redpanda.KafkaAPI,  addlListeners)

It also appears that config.Set has been removed (or at least I can't find it) in newer versions of rpk 🤔

[paulzhang97]

I see config.set is still there.

I will take a look at replacing config.Set, and see whether it is viable since we would not want to change the way how the env ADDITIONAL_LISTENERS is set.

FYI, if using double quotes instead of single quotes, I get the error such as invalid character 'n' after object key:value pair.

[paulzhang97]

Update: it works if I use escaped double quotes as the outer string uses double quotes, like, "[{\"name\": \"mtls-kafka\", 'address': '{{ .Index }}-f415bda0-{{ .HostIP | sha256sum | substr 0 7 }}.redpanda.com', 'port': {{39002 | add .Index}}}]"

[paulzhang97]

I think I did it. Please check.

[chrisseto]

This entire file would dramatically benefit from some unit testing. It may also be useful to execute the template and unmarshal the resultant value to ensure everything's being formatted correctly.

If the previously mentioned custom types work, you could narrow down the testing scope to just those types.

[paulzhang97]

The new unit tests added for statefulset exercise the functions. It is confusing that it seems no test coverage for this file. I will try to do some refactoring or create extra tests.

[paulzhang97]

Added new unit tests.

[chrisseto]

Don't mutate global values. There's a package that exports a Scheme with both v1 and v2 CRs added that you can reference instead of building a new scheme here.

[paulzhang97]

I have not found the pkg so far. Could you send a link to it?

[paulzhang97]

I think I found it. Is it this?

[chrisseto]

That's the one!

[chrisseto]

Does require.ElementsMatch not work here?

[paulzhang97]

It works. TIL.

[chrisseto]

Again citing the above comment, you should be able to ditch all of this if you have a type that can correctly serialize templated values.

[paulzhang97]

I know. I will think hard to see whether we can leverage customize MarshalJSON functions.

[chrisseto]

TODO what?

[paulzhang97]

I will remove.

[paulzhang97]

Done.