[Vector, VictoriaLogs, Bareos] Use VRL and remap function to create a parser in order to sink Bareos logs into VictoriaLogs over HTTP

[Rawoosa]

OS: Rocky Linux 9.6
Vector: Docker, latest-alpine

Let's imagine we have a Bareos log file located at /var/log/bareos/bareos.log with 2 lines like this:
Line 1: 12-Sep 15:56 bareos-dir JobId 40: Version: 24.0.5~pre32.7c5f79a1e (14 August 2025) Red Hat Enterprise Linux release 9.6 (Plow)
Line 2: 12-Sep 15:56 bareos-dir JobId 40: Start Backup JobId 40, Job=DefaultBackup.2025-09-12_15.56.27_23

In vector.yaml we add a source section with this file:

sources:
  input-bareos:
    type: file
    include:
      - "/var/log/bareos/bareos.log"
    ignore_older: 86400
    read_from: end

After that we'd like to extract multiple fields in order to filter logs in VictoriaLogs. We do it using transform section, remap function and regex:

transforms:
  parser-bareos:
    type: remap
    inputs:
      - input-bareos
    source: |
      . = parse_regex!(.message, r'^(?P<timestamp>\d{2}-[A-Za-z]{3} \d{2}:\d{2}) (?P<component>[a-zA-Z0-9._-]+) JobId (?P<jobid>\d+): (?P<msg>.*)$')

      .timestamp = parse_timestamp!(.timestamp, "%d-%b %H:%M")
      .jobid = to_int!(.jobid)

Finally, we sink the result into Vlogs using HTTP:

sinks:
  vlogs-bareos:
    type: http
    inputs:
      - parser-bareos
    uri: "http://172.16.110.3:9428/insert/jsonline?_stream_fields=stream,component,jobid&_msg_field=msg&_time_field=timestamp"
    encoding:
      codec: json
    compression: gzip
    framing:
      method: newline_delimited
    healthcheck:
      enabled: false

In result, Vector seems to sink only the message field from the input file and do not actually get required fileds using the parser:

I believe I've failed to correctly set up a parser and I require some help. Thanks in advance!

[pront]

Hi @Rawoosa, I recommend attaching a console sink to help you debug further and understand the shape of events that reach the sink:

    console:
      type: console
      inputs: [ "parser-bareos" ]
      encoding:
        codec: json
        json:
          pretty: true

Also, vector tap is another helpful debugging tool to inspect what comes out of the remap transform.

[Rawoosa]

Looking at logs I see three fileds being sent: file, host and message.
It's the message field I want to split into multiple ones in order to filter them in VLogs.
Unfortunately I can't exactly wrap my head around how do I fix the parser I already have to achieve this, I followed examples in the documentation but I'm sure I've missed smth. Would you please point it out?

[pront]

A general note, it is hard to understand what you want to achieve based on this description and the screenshots. Please share sample logs in markdown format. An actual log sample vs what the desired one would be very helpful here.

It's the message field I want to split into multiple ones in order to filter them in VLogs.

I recommended debugging your regex using the VRL playground: here

[Rawoosa]

My appologies, I don't have much experience in discussing such topics :D

Let's try this. These are logs from Bareos:

12-Sep 15:56 bareos-dir JobId 40: Version: 24.0.5~pre32.7c5f79a1e (14 August 2025) Red Hat Enterprise Linux release 9.6 (Plow)
12-Sep 15:56 bareos-dir JobId 40: Start Backup JobId 40, Job=DefaultBackup.2025-09-12_15.56.27_23
12-Sep 15:56 bareos-dir JobId 40: Connected Storage daemon at prcrd-test-bareos-server.rawoochat.org.ua:9103, encryption: TLS_AES_256_GCM_SHA384 TLSv1.3
12-Sep 15:56 bareos-dir JobId 40:  Encryption: TLS_AES_256_GCM_SHA384 TLSv1.3
12-Sep 15:56 bareos-dir JobId 40: Connected Client: test-client at 172.16.110.2:9102, encryption: TLS_AES_256_GCM_SHA384 TLSv1.3
12-Sep 15:56 bareos-dir JobId 40:  Handshake: Immediate TLS 
12-Sep 15:56 bareos-dir JobId 40:  Encryption: TLS_AES_256_GCM_SHA384 TLSv1.3
12-Sep 15:56 bareos-sd JobId 40: Using just in time reservation for job 40
12-Sep 15:56 bareos-dir JobId 40: Using Device "JustInTime Device" to write.
12-Sep 15:56 prcrd-test-bareos-client.rawoochat.org.ua-fd JobId 40: Connected Storage daemon at prcrd-test-bareos-server.rawoochat.org.ua:9103, encryption: TLS_AES_256_GCM_SHA384 TLSv1.3
12-Sep 15:56 prcrd-test-bareos-client.rawoochat.org.ua-fd JobId 40:  Encryption: TLS_AES_256_GCM_SHA384 TLSv1.3
12-Sep 15:56 bareos-sd JobId 40: Version: 24.0.5~pre32.7c5f79a1e (14 August 2025) Red Hat Enterprise Linux release 9.6 (Plow)
12-Sep 15:56 prcrd-test-bareos-client.rawoochat.org.ua-fd JobId 40: Extended attribute support is enabled
12-Sep 15:56 prcrd-test-bareos-client.rawoochat.org.ua-fd JobId 40: ACL support is enabled
12-Sep 15:56 prcrd-test-bareos-client.rawoochat.org.ua-fd JobId 40: Version: 24.0.5~pre32.7c5f79a1e (14 August 2025) Red Hat Enterprise Linux release 9.6 (Plow)
12-Sep 15:56 bareos-sd JobId 40: Because no backup data was received, no device was reserved. As such no Session Labels were written for this job.
12-Sep 15:56 bareos-sd JobId 40: Elapsed time=00:00:01, Transfer rate=0  Bytes/second

They all start with the same Timestamp, after which comes the component of Bareos which sent this log, then comes JobId text with the actual JobID number and then finally comes the message of the log.

So from the message field of the input I want to get these fields:

1. .timestamp -- Date and time of the log (For example, 12-Sep 15:56);
2. .component -- Bareos component (For example, bareos-dir ot test-client-fd);
3. .jobid -- The ID of the Bareos job (For example, 40);
4. .msg -- the actual message of the log, everything else that comes after the jobid and semicolon (For example, Elapsed time=00:00:01, Transfer rate=0 Bytes/second)

Let's take this log line as an example
12-Sep 15:56 bareos-sd JobId 40: Elapsed time=00:00:01, Transfer rate=0 Bytes/second
In the end I want to receive smth like this:
{"timestamp":<timestamp, I'll tweak it myself>,"component":"bareos-sd","jobid":40,"msg":"Elapsed time=00:00:01, Transfer rate=0 Bytes/second"}

Please let me know if this is sufficient or if you need more info. I'll try using the VRL playground meanwhile.

[Rawoosa]

Alrighty, so it seems the correct function would be parse_regex_all instead of just parse_regex. Now I need to figure out how to fix 400 bad request.

[pront]

parse_regex is the right function for one log line. The failure in the original program is parse_timestamp: %d-%b %H:%M does not supply a year. With the default drop_on_error: false, the failed remap forwards the original event, which explains the missing parsed fields.

This parses all 17 sample lines. The year and timezone are explicit because the log does not contain them; 2025 and UTC are example values to replace with the correct values for your logs.

. = parse_regex!(.message, r'^(?P<timestamp>\d{2}-[A-Za-z]{3} \d{2}:\d{2}) (?P<component>[a-zA-Z0-9._-]+) JobId (?P<jobid>\d+): (?P<msg>.*)$')
.timestamp = parse_timestamp!("2025 " + .timestamp, format: "%Y %d-%b %H:%M", timezone: "UTC")
.jobid = to_int!(.jobid)

Try the complete program in the VRL playground.

Please don't hesitate to followup in this thread if you have further questions.

[Rawoosa]

You're an absolute legend.

Not only I forgot I even started this thread, I left the workplace where I was messing around with Vector almost a year ago.
Much respect for remembering and providing a solution with detailed explanation.
Thank you!