ECS metadata fails when container tag is overriden

[RaJiska]

Bug Report

Describe the bug
ECS metadata plugins fails to retrieve task details when the tag is overridden (by default the container ID) for something custom.

To Reproduce
Config file:

[SERVICE]
  daemon    Off
  log_level info
  flush     30

[INPUT]
  Name   forward
  Listen 0.0.0.0
  Port   ${AGENT_PORT}

[FILTER]
  Name                           ecs
  Match                          ecs.*
  ADD ecs_cluster                $ClusterName
  ADD ecs_task_id                $TaskID
  ADD ecs_container_name         $ECSContainerName
  ADD ecs_task_definition_family $TaskDefinitionFamily
  ADD ecs_task_id                $TaskID

[OUTPUT]
  Name  stdout
  Match ecs.*

Container sending logs started with the following command:

docker run --rm -it --log-driver=fluentd --log-opt fluentd-address=localhost:24224 --log-opt "tag=ecs.test123456" alpine echo lol

Resulting in the following errors from fluentbit:

[2024/07/01 09:38:49] [ warn] [filter:ecs:ecs.1] Failed to get metadata from /v1/tasks?dockerid=ecs.test1234, will retry
[2024/07/01 09:38:49] [ info] [filter:ecs:ecs.1] Requesting metadata from ECS Agent introspection endpoint failed for tag ecs.test123456

Expected behavior
Fluent-bit ECS filter should not retrieve the container ID from the tag. I am not aware of the internals, but I believe there must be another way to retrieve the container ID as it is available through the container_id key regardless.

Your Environment

• Version used: fluent/fluent-bit:3.0.7
• Configuration: Described above
• Environment name and version (e.g. Kubernetes? What version?): docker
• Filters and plugins: ecs

Additional context
I'd like to supply a custom tag to have a different logic depending of the container and log types it ships to fluentbit. The ECS metadata filter does not work when supplying my own tag.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[jenademoodley]

Came across this same issue during testing. I was able to use custom tags by specifying {{.FullID}} after the custom tag to add the container ID to the tag:

    "logConfiguration": {
        "logDriver": "fluentd",
        "options": {
            "fluentd-address": "localhost:24224",
            "tag":"nginx-service.{{.FullID}}"
        }
    }

The filter would then need to have the above tag as an ECS tag prefix:

[FILTER]
    Name ecs
    Match nginx-service.*
    ECS_Tag_Prefix nginx-service.
    ADD container_instance_arn $ContainerInstanceArn
    ADD ecs_task_arn $TaskARN
    ADD cluster $ClusterName

Only pain point is that there needs to be a filter for each custom tag you specify

[RaJiska]

@jenademoodley Unfortunately this is not good enough as there are cases where you may have tags looking like nginx-service.{{.FullID}}, and nginx-service.typeX,{{.FullIID}}.
You can't target just the nginx-service.{{,FullID}} tag as you necessarily have to use a wild card to match the unknown FullID, and matching nginx-service.typeX.{{.FullID}} as an unintended side effect.

I opened a PR #9033 that addresses just that by providing a field to lookup the container ID from. Unfortunately this hasn't been reviewed yet.