monitoring file completion status in file-to-Kafka pipelines

[whyjp]

When using Vector for a file-to-Kafka pipeline, is there a way to configure a dashboard or check metrics to confirm that file uploads are complete/done?

Due to the complexity of various rotation/rolling environments and real-time tailing issues, the pairing of filename:offset may not match the metrics:offset of transmitted logs.

Based on the current "file EOF reached" metrics content, it's difficult to make an accurate determination.

Is there any good method to precisely determine when a file is done - monitoring stopped - and no longer being written to?

[pront]

Hi @whyjp, the file source emits the open_files gauge metric (which is unfortunately not documented). Is that what you are looking for?

[whyjp]

I know that -open_files
it is just file monitor begin but I want to know rich to eof of files each files

I can see that it started with the current metric, but it's hard to know that it's been handled properly and completely to the end.

[pront]

You're right: open_files and reaching EOF do not confirm that a file has been completely delivered to Kafka. The file source may receive more appended data, while end-to-end acknowledgements track delivery separately.

There is no built-in per-file metric that combines producer completion with Kafka delivery confirmation. #23558 requests renaming files after EOF; if you need a completion metric tied to sink acknowledgements, please open a feature request.

Please don't hesitate to followup in this thread if you have further questions.