Configcheck secret fails to reconcile when there is too much data in the reconcile for fluentd

[mostrovoi]

I have a setup that creates around 200 flows from flow CRD in a cluster and when applying the changes that go from a centralized clusterflow to 200 flows I get the following error:

erator-logging","reconcileID":"68addd32-12a5-4d44-9e08-e539187b0ba1","error":"failed to validate config: failed to create secret for fluentd configcheck: Secret "logging-operator-logging-fluentd-configcheck-c308675f" is invalid: data: Too long: must have at most 1048576 bytes","errorVerbose":"Secret "logging-operator-logging-fluentd-configcheck-c308675f" is invalid: data: Too long: must have at most 1048576 bytes\nfailed to create secret for fluentd configcheck\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentd.(*Reconciler).configCheck\n\t/workspace/pkg/resources/fluentd/appconfigmap.go:83\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentd.(*Reconciler).Reconcile\n\t/workspace/pkg/resources/fluentd/fluentd.go:168\ngithub.com/kube-logging/logging-operator/controllers/logging.

I suspect that the issue is that the created secret exceed 1Mb which is the max size for a secret creation in k8s when the logging-operator goes through the newly defined flows. What would be the workaround for this?

One way to go about this is to do batches of 10-20 flows and then reconcile would work I guess. Is there any other way?
Thanks

Version of logging-operator is 4.2.2

[pepov]

there is a compressConfigFile flag in the fluentd spec, hopefully 4.2.2 supports it, but it is recommended to upgrade anyways, since neither the logging operator code, neither the images are supported with that version.

[mostrovoi]

quick update. I disabled the config check in the operator like this:

flowConfigCheckDisabled: true

but then I m getting this error:
failed to reconcile resource: updating resource failed: Secret "logging-operator-logging-fluentd-app" is invalid: data: Too long: must have at most 1048576 bytes","errorVerbose":"Secret "logging-operator-logging-fluentd-app" is invalid: data: Too long: must have at most 1048576 bytes\nupdating resource f

So the logging operator uses yet again a new secret to store all new fluentd.conf and the total of flows and the final fluentd.conf exceeds the max size of 1mb. 2 questions related to this

a) Would the compressConfigFile help me with this situation?
b) Is the secret logging-operator-logging-fluentd-app storing the fluentd.conf delta or the final fluentd.conf? The reason I am asking is if it is storing the delta (data to be reconciled only) then I could create the new flows in batches of new deployments and have few reconcile operations rather than a large one. If, however, the fluentd.conf stores the final configuration I dont know how I could overcome this one

Thanks!

[mostrovoi]

So if I get this right, the problem is a limit in the number of flows that the operator logging can handle? In my tests, when creating more than 200 flows the final fluentd.conf exceeds the 1Mb size and then fails to create the secret for that

What am I missing here? Does the logging-operator need to create a secret containing the final fluentd.conf always? Is there any workaround for this?

[pepov]

a) Would the compressConfigFile help me with this situation?

It should help, yes. However we received a report from a different user, that it might have a bug around that. We can look at that bug if that's the case, but it would be good to know if it worked for you or not.

b) Is the secret logging-operator-logging-fluentd-app storing the fluentd.conf delta or the final fluentd.conf?

Always the final config.

An alternative solution is to shard your single fluentd instance into multiple ones using FluentdConfigs, where you can create separate "tenants" that have their isolated configuration. Please reach out over discord or slack if you are interested to learn more:
https://kube-logging.dev/docs/community/

[csatib02]

Hey @mostrovoi,
Just checking in on this one, where you able to fix this based on: #1944 (comment)?

[csatib02]

Closing this.
Let us know if you require further assistance!