Teragrep Record Schema Mapper Library for Java

Rsm_01 is a library that provides record schema mapping (log normalization) functionality to Java using Liblognorm library that is written in C-code. All the functionalities of the Liblognorm library are provided to Java through the use of Java Native Access (JNA) library.

Features

• Provides full access to log normalization functionalities of Liblognorm C-library in Java, which in short allows:

  1. Matching a message line to a rule from provided rulebase configuration.

  2. Picking out variable fields from the line according to the configuration.

  3. Returning the picked variable fields as a JSON string object.

• Capability to handle errors occurring in the C-library through Java exceptions.

How to compile

Maven and liblognorm-devel must be installed to build the project.

Project building has been tested on Fedora 41.

dnf install liblognorm-devel

mvn clean package

How to use

Configuration

To use rsm_01 for log normalization, two things are required:

1. Log messages

2. A rulebase, which configures how the messages are normalized by liblognorm

The rulebase consists of lines of text where each line reflects the structure of your log messages. Liblognorm will generate a parse-tree from the rulebase and then use it to parse the log messages during normalization. The provided rulebase must follow the version two configuration guidelines described in the liblognorm documentations.

A simple rulebase file with two different rules is structured as follows:

version=2
rule=:%all:rest%
rule=tag:Amount: %N:number%

Normalization

Normalization is done by first initializing LognormFactory object with the rulebase given as an input argument for the constructor. The lognorm() method of the factory object is then used to provide a configured version of JavaLognormImpl object that is capable of normalizing messages through the normalize() method.

The rulebase can be provided to the LognormFactory constructor either as a String or as a File object. Rulebases in String format do not require the version=2 indicator mentioned in liblognorm documentations.

Because of C memory management requirements, the JavaLognormImpl object must be closed when it is no longer needed. The typical use-case for normalization using try-with-resources is as follows:

LognormFactory lognormFactory = new LognormFactory("rule=:%all:rest%");
try (JavaLognormImpl javaLognormImpl = lognormFactory.lognorm()) {
    String normalizedMessage = javaLognormImpl.normalize("message to normalize");
}

If normalization was a success, normalizedMessage will hold the normalization result in JSON string format.

Additional configuration options

Liblognorm API has additional configuration options available for normalization:

1. LN_CTXOPT_ADD_ORIGINALMSG — always adds original message to the normalization result.

2. LN_CTXOPT_ADD_RULE — adds mockup rule to the normalization result.

3. LN_CTXOPT_ADD_RULE_LOCATION — adds rule location (file, lineno) to the normalization result.

The configuration for the additional options is done by providing OptionsStruct object as an additional input argument for LognormFactory constructor.

LibJavaLognorm.OptionsStruct opts = new LibJavaLognorm.OptionsStruct();
opts.CTXOPT_ADD_ORIGINALMSG = true;
LognormFactory lognormFactory = new LognormFactory(opts, "rule=:%all:rest%");
try (JavaLognormImpl javaLognormImpl = lognormFactory.lognorm()) {
    String normalizedMessage = javaLognormImpl.normalize("message to normalize");
}

Contributing

You can involve yourself with our project by opening an issue or submitting a pull request.

Contribution requirements:

1. All changes must be accompanied by a new or changed test. If you think testing is not required in your pull request, include a sufficient explanation as why you think so.

2. Security checks must pass

3. Pull requests must align with the principles and values of extreme programming.

4. Pull requests must follow the principles of Object Thinking and Elegant Objects (EO).

Read more in our Contributing Guideline.

Contributor License Agreement

Contributors must sign Teragrep Contributor License Agreement before a pull request is accepted to organization’s repositories.

You need to submit the CLA only once. After submitting the CLA you can contribute to all Teragrep’s repositories.