How to realize a sequence count on message (stream) - focus for auditing?

[drgsr]

Question

Hello everybody,
we are currently integrating vector as a cool logging filter for dedicated purposes - in our case focus for "audit-logs".
So we need some (simple) solution for the questions:

• Has somebody edited the audit.log file? (e.g. with an editor removed lines)
• Are there audit-messages lost, e.g. as feeding to a cloud service? (e.g. in the network transfer)

One simple solution were a sequence count...
In both cases, it is fairly simple to check, whether all numbers within the sequence-cont field are covered or, (if the timely sorting is preserved) is the next sequence-cont value an increment from the predecessing one?

Setting:
We read from set of sources, which are filtered and sent into one (audit) stream, which at the end of the day is stored in some audit.log file.
Some sort of SEQUENCE_COUNT is already realized by systemd.journald for its messages, but: since we are filtering for a subset, AND also attach to other sources, that SEQUENCE_COUNT is not meaningful anymore.

Well - I do understand, that vector is functional oriented (which I really like as paradigm), and therefore does not like such interdependence (some sort of global variable). But for our audit use-case it is a sort of solution.

Potential solutions

• I could imagine to solve this question with a dedicated (new) transform, which has the only purpose to generate a sequence count. The counter state could then be stored within the context, and survive also a vector restart to be continued without interruption.

• Alternatively, if one could syntactically access "per message" the metrics of the remap transform for the counter component_received_events_total via vrl, the issue could be solved that way. Unfortunately I am not aware of this kind of access.

Sketch: In the "remap" transform a formal access to the metrics of the own instance, like:

.seq_count=self.component_received_events_total
https://vector.dev/docs/reference/configuration/transforms/remap/#component_received_events_total

Any ideas?

Thanks in advance,
DrG

Vector Config

No response

Vector Logs

No response

[pront]

Vector does not currently provide a built-in sequence generator with durable state across restarts.

One idea that might work for your audit stream is to assign sequence numbers in a durable producer or log store after filtering. Whether that fits would depend on how your sources are combined and where you need to detect missing events.

The component metric would not be suitable for numbering individual events. Although Vector now has get_vector_metric, it reads a periodically refreshed snapshot.

You are welcome to open a feature request for a persistent sequence counter in Vector.

Please don't hesitate to followup in this thread if you have further questions.