Environment variables are not resolved when using templates

[massive]

It seems that environment variables are not resolved the same way in templates as they are in regular configuration files. There is a workaround, so it's not a major issue, but I assume this could be a challenge for many newcomers, as it was for me.

Given this template:

type: input
name: kafka
fields:
  - name: topic
    type: string

mapping: |
  #!blobl
  root = {
    "kafka_franz": {
      "seed_brokers": [ "${KAFKA_BROKERS}" ],
      "topics": [ this.topic ],
      "sasl": ...
    }
}

Here's the error I encounter when using templates:

WARN unable to open connection to broker           @service=redpanda-connect addr="${KAFKA_BROKERS}:9092" broker=seed_0 err="dial tcp: lookup ${KAFKA_BROKERS}: no such host" label="" path=root.input
(edited)

It's simple to replace seed_brokers with env("KAFKA_BROKERS"), if you know you need to, but this is a bit of a gotcha.

Moreover, it seems you can't set defaults using environment variables. Both default: "${SOME_VARIABLE}" and default: ${! env("SOME_VARIABLE") } appear to evaluate to null, resulting in the error: expected object value, got !!null.

[mihaitodor]

Thanks for raising this @massive!

Moreover, it seems you can't set defaults using environment variables. Both default: "${SOME_VARIABLE}" and default: ${! env("SOME_VARIABLE") } appear to evaluate to null, resulting in the error: expected object value, got !!null.

For default values, when ${SOME_VARIABLE} is evaluated, you should be able to do ${SOME_VARIABLE:foobar} (${SOME_VARIABLE:} if you just want an empty string). In the case of the env() bloblang function, you can do ${! env("SOME_VARIABLE").or("foobar") }.

[massive]

I've also noticed that testing templates do not support setting env variables similarly to regular configuration.