Unexpected re-processing of old log files (30h+) after OOM crash despite ignore_older_secs: 86400 and successful S3 delivery

[ldongyeop2]

Question

Hi Vector team,

I am experiencing an unexpected behavior where Vector re-processes and re-uploads old log files to S3 after a pod crash, even though those files were successfully uploaded hours before the crash and should have been filtered out by ignore_older_secs.

Environment:

Platform: EKS (Amazon Elastic Kubernetes Service)

Vector Version: 0.53.0

Vector Role: Log collector (Agent)

Source: file

Sink: aws_s3

Timezone: KST (UTC+9)

Incident Timeline (KST):

Mar 16, 09:00 - 16:00: Vector was running normally. Logs from the morning (Mar 16 AM) were successfully read and uploaded to S3.

Mar 16, 16:00: The EKS node and Vector pod crashed (OOM Killed).

Mar 17, 17:00: The pod was finally restored after increasing memory limits.

Observation: Upon restoration, Vector re-read and re-uploaded Mar 16 AM logs (approx. 31 hours old at the time of restoration) from the beginning, causing significant duplicates in S3/Databricks.

The Mystery:
At the time of restoration (Mar 17, 17:00 KST), the files from Mar 16, 09:41 KST were 31+ hours old.

Why did the file source pick up these files despite ignore_older_secs: 86400?

If the files were successfully uploaded before the crash, why did Vector lose the checkpoint or decide to re-read from the beginning (position: 0)?

I suspect this might be related to Issue #17208, where checkpoints are expired if they are older than ignore_older_secs during startup. However, I don't understand why these files bypassed the "discovery" filter in the first place if they were already older than 24 hours.

Could you help me understand why Vector ignores the 24h filter for these specific files and why the previous checkpoints were not honored or were deleted?

Vector Config

customConfig:
data_dir: "${MY_VECTOR_DATA_PATH}"

sources:
game_ndjson:
type: file
include:
- "/var/server_log/game_dev_kr/databricks-*"
ignore_older_secs: 86400
fingerprint:
strategy: "device_and_inode"
line_delimiter: "\n"

transforms:
filter_fields:
type: "remap"
inputs:
- "game_ndjson"
source: |
path = string!(.file)
parts = split(path, "/")
filename = parts[-1]
.src_filename = filename

    del(.host)
    del(.timestamp)
    del(.source_type)
    del(.file)

    parsed_data, err = parse_json(.message)
    if err == null {
        . = merge!(., parsed_data)
        del(.message)
    } else {
        .tags.parse_error = true
        log("json parsing error")
    }

sinks:
s3_archive:
type: "aws_s3"
inputs:
- "filter_fields"
bucket: "amzn-s3-databrix-bucket"
region: "ap-northeast-2"
key_prefix: "service=gameserver/%Y-%m-%d/{{ {{ }} src_filename }}-"
compression: "gzip"
acknowledgements:
enabled: true

  # 실패시 request
  request:
    # 동시성 조절
    # concurrency: 2
    # 실패시 최대 10번까지 시도
    retry_attempts: 10
    # error 발생시 2초 쉬고 다시 시도
    retry_initial_backoff_secs: 2
    # 계속해서 error 발생시 최대 30초까지 쉬면서 시도
    retry_max_duration_secs: 30

  encoding:
    codec: "json"
  buffer:
    type: "disk"
    # must be greater than or equal to 268435488 오류로 크기 확장
    max_size: 772629760
    when_full: "block"
  batch:
    max_bytes: 20971520
    timeout_secs: 600

Vector Logs

No response

[pront]

ignore_older_secs uses the file's modification time, not the timestamps inside its log records or filename. A file containing 31-hour-old records can still qualify if it was modified more recently.

Your suspicion about #17208 is plausible. To help us investigate further, could you share:

• The affected file's modification time at restart, and whether it was appended to, copied, or restored during the outage.
• Whether ${MY_VECTOR_DATA_PATH} persisted across the node/pod replacement.
• Startup logs showing checkpoint loading and file positions, along with any changes to the file's device/inode identity.

Please don't hesitate to followup in this thread if you have further questions.