Gather additional docker container metadata using signoz-otel-collector?

[rwjack]

I'm testing the signoz-otel-collector on a docker host, and I'm trying to get information about all other docker containers running on that host. The logs are flowing in nicely to signoz, but I can't seem to get the following attributes from other containers:

• hostname
• image name
• image tag

Only the container_id (uuid) is logged, which doesn't provide much observability.

Any thoughts on configuring additional metadata logging for docker?

version: "3"

services:

  collector:
    image: signoz/signoz-otel-collector:0.66.5
    restart: unless-stopped

    container_name: signoz-otel-collector
    hostname: signoz-otel-collector

    user: root:root # can't read /var/lib/docker/containers dir without this

    volumes:
      - ./otel-collector-config.yaml:/etc/otel/config.yaml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro

receivers:
  filelog/containers:
    include: [  "/var/lib/docker/containers/*/*.log" ]
    start_at: end
    include_file_path: true
    include_file_name: false
    operators:
    - type: json_parser
      id: parser-docker
      output: extract_metadata_from_filepath
      timestamp:
        parse_from: attributes.time
        layout: '%Y-%m-%dT%H:%M:%S.%LZ'
    # Extract metadata from file path
    - type: regex_parser
      id: extract_metadata_from_filepath
      regex: '^.*containers/(?P<container_id>[^_]+)/.*log$'
      parse_from: attributes["log.file.path"]
      output: parse_body
    - type: move
      id: parse_body
      from: attributes.log
      to: body
      output: add_source
    - type: add
      id: add_source
      field: resource["source"]
      value: "docker"
    - type: remove
      id: time
      field: attributes.time
processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 11000
    timeout: 10s
  resourcedetection:
    detectors: ["docker"]
exporters:
  otlp/log:
    endpoint: http://signoz.sec.arpa:4317
    tls:
      insecure: true
service:
  pipelines:
    logs:
      receivers: [filelog/containers]
      processors: [batch, resourcedetection]
      exporters: [ otlp/log ]

[nityanandagohain]

Hi, the default filedlog receiver can access other metadata attributes as of now. You will have to use another tool for that. You can check this out #1597 (comment)

[rwjack]

Interesting find, though it just presents another dependency to be maintained and configured.

Imagine I come to your shop for vanilla/chocolate ice cream. You have both flavors, but in order to actually buy them, you tell me I have to go 2 blocks further for an ice cream cone. 🍦

Not really a proper solution, is it? 😄

[nityanandagohain]

Yeah, I agree that it is not a proper solution, but until we don't have something like k8sattributes processor in opentelemetry for docker, through which can enrich the metadata of logs, we will have to rely on another dependency.

[rwjack]

Well, since we have attributes["log.file.path"], would there be a way to do something like:

example attributes["log.file.path"] = /var/lib/docker/containers/abc1234/abc1234-json.log
---
get dirname of attributes["log.file.path"]   # /var/lib/docker/containers/abc1234
read container_hostname                      # /var/lib/docker/containers/abc1234/hostname
add attribute["container_hostname"]          # test-container

[rwjack]

open-telemetry/opentelemetry-collector-contrib#18920