Logs are missing in OpenSearch for a while, how to check and fix?

[Paul-AWS]

I can't see logs in OpenSearch for a while, how to find out the root cause and fix it?

[Paul-AWS]

You may use monitoring and logs feature in Centralized Logging with OpenSearch pipelines to locate the root cause.

The log data flow: Log source (Services/Applications) -> S3 -> SQS -> Lambda -> OpenSearch.

Step 1. Check the Monitoring metrics

Log into the Centralized Logging with OpenSearch console, and click into the pipeline. Choose the "Monitoring" Tab, check the metrics.

In Buffer part:

1. If the value of "Number of Messages Received" is 0, this means there are no files on S3, or SQS was not successfully triggered. Please check the S3 and SQS.
2. If the value of "Number of Messages Received" is smaller than the value of "Number of Messages Deleted", this means Lambda was not successfully triggered. Please check the limit of Lambda. (Concurrent Execution)
3. If the value of "Number of Messages Received" is equal to the value of "Number of Messages Deleted", this means S3 and SQS works well. Please check the metrics below.

In Lambda Processor part, focus on these two metrics: "Number of logs failed to process" and "Error count". If the value of these two metrics is not 0, please check the processor logs for more details.

Step 2. Check the Processor logs

Switch to the "Logs" tab, and click the link in Log of processor lambda.

You may filter the logs by timestamp and the keyword "error". Then locate the root issue based on the error information in the log and take appropriate countermeasures. If the error message shows the Lambda failed to write logs into OpenSearch, you need also check the OpenSearch status.

Step 3. Check the OpenSearch

In OpenSearch service console, check the metrics of related OpenSearch cluster. If the cluster is in Yellow or Red status, and the metrics show that there are insufficient resources like CPU, memory or disks, you may need to make configuration changes such as add nodes, upgrade node instance type, or increase disk capacity.