Management of null values on indices

[AlexRuiz7]

Description

Avoid indexing void data, such as empty strings, using null instead. Null values can become searchable by defining a replacement on the index mappings. See https://opensearch.org/docs/latest/field-types/supported-field-types/index/#null-value

By default, OpenSearch replaces null values with a hyphen, for better human-readable format on visualizations and dashboards. The field is shown a null in the JSON representation of the document. The default replacement (hyphen) can be customized per field, allowing us to add any default value we see fit when there is no data for such field.

In the image below, we can see how fields using the null value are shown with a hyphen, making it clear that there is no value for such field, while fields whose value uses an empty string show nothing, which tend to be misleading as one could think something is broken in the visualization.

The goal of the issue is to ensure that any value indexed follows these criteria:

• No empty values allowed.
• When there is no value for such a field, null is used instead.

Functional requirements

• The Wazuh Agent sends data to the Indexer, through the Wazuh Server, with no empty values.
• The Wazuh Engine sends data to the Indexer, through the Indexer Connector, with no empty values.
• The Wazuh Server sends data to the Indexer, through the API on port 9200, with no empty values.
• Every component sending data to the Indexer ensures that empty values, if present, are properly replaced with null beforehand.

Implementation restrictions

• We are referring explicitly to the null value, not the "null" string.
• By empty values, we refer to empty instances of:
  • String: "" --> null.
  • Arrays: [] --> null or [null].
• Empty values must either be replaced by null or not indexed.

Plan

• Validation of data sent by the Agent to replace empty values.
  • Owner: @wazuh/devel-xdrsiem-agent
  • Teams involved: @wazuh/devel-xdrsiem-indexer @wazuh/devel-xdrsiem-server @wazuh/devel-xdrsiem-dashboard
• Validation of data sent by the Engine to replace empty values.
  • Owner: @wazuh/devel-xdrsiem-server
  • Teams involved: @wazuh/devel-xdrsiem-indexer @wazuh/devel-xdrsiem-dashboard
• Validation of data sent by the Server to replace empty values.
  • Owner: @wazuh/devel-xdrsiem-server
  • Teams involved: @wazuh/devel-xdrsiem-indexer @wazuh/devel-xdrsiem-dashboard
• Proposal of null_value parameter in the index’s mappings
  • Owner: @wazuh/devel-xdrsiem-indexer
  • Teams involved: @wazuh/devel-xdrsiem-indexer @wazuh/devel-xdrsiem-dashboard @wazuh/devel-xdrsiem-server @wazuh/devel-xdrsiem-agent