Skip to content

[AGNTLOG-229 Multiline features FAQ #29931

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions content/en/agent/logs/advanced_log_collection.md
Original file line number Diff line number Diff line change
Expand Up @@ -629,6 +629,21 @@ All the logs collected by the Datadog Agent are impacted by the global processin

**Note**: The Datadog Agent does not start the log collector if there is a format issue in the global processing rules. Run the Agent's [status subcommand][6] to troubleshoot any issues.

## Multi-line log aggregation FAQ

**1. When should I use manual multi-line rules vs. automatic multi-line detection?**

If you know the format of your logs, you should use manual multi-line rules for more precise control.
If you are sending lots of multi-line logs and you are unsure of their format or don't have the means to configure all sources individually, you should use automatic multi-line detection.
Comment on lines +636 to +637
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you know the format of your logs, you should use manual multi-line rules for more precise control.
If you are sending lots of multi-line logs and you are unsure of their format or don't have the means to configure all sources individually, you should use automatic multi-line detection.
If you know the format of your logs, you should use manual multi-line rules for precise control.
If you are sending lots of multi-line logs, and you are unsure of their format or don't have the means to configure all sources individually, you should use automatic multi-line detection.


**2. What happens when a multi-line pattern doesn't match any logs?**

All non-JSON log lines will be processed individually as separate log entries.
All JSON-formatted log lines will be treated as a single line of logs, and only the first valid JSON format will enter the intake; the rest will be dropped.
Comment on lines +641 to +642
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
All non-JSON log lines will be processed individually as separate log entries.
All JSON-formatted log lines will be treated as a single line of logs, and only the first valid JSON format will enter the intake; the rest will be dropped.
All non-JSON log lines are processed individually as separate log entries.
All JSON-formatted log lines are treated as a single line of logs, and only the first valid JSON format enters the intake; the rest are dropped.


**3. What happens when there are both global rules and integration-specific rules?**
Integration-specific rules completely override global rules for the particular integration.

## Further Reading

{{< partial name="whats-next/whats-next.html" >}}
Expand Down
2 changes: 1 addition & 1 deletion content/en/agent/logs/auto_multiline_detection.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ Tokens include
- special characters
- datetime components.

Each log token is compared to each token in the sample. If 75% of the logs tokens match the samples, the log is marked for aggregation.
Each log token is compared to each token in the sample. If 75% of the log's tokens match the sample's, the log is marked for aggregation.
Datadog recommends using sample-based matching if your logs have a stable format. If you need more flexible matching, you can use regex.

### Regex patterns
Expand Down
Loading