Merge pull request #19381 from newrelic/NR-342335-aws-log-forward-new…

…-integration

feat (AWS): Added guided steps with restrictions and troubleshooting to integrate AWS with NR for log forwarding

src/content/docs/infrastructure/amazon-integrations/get-started/introduction-aws-integrations.mdx

@@ -23,11 +23,11 @@ redirects:
 freshnessValidatedDate: 2024-09-20
 ---
 
-New Relic offers several methods to collect performance data about your Amazon Web Services (AWS) environment. Once you've connected New Relic with AWS, you can access your system's AWS telemetry, letting you closely monitor the health of your AWS stack. This doc describes how to get started with AWS monitoring and provides an overview about AWS costs. 
+New Relic offers several methods to collect performance data about your Amazon Web Services (AWS) environment. After you've connected New Relic with AWS, you can access your system's AWS telemetry, letting you closely monitor the health of your AWS stack. Our AWS observability solution supports [Amazon CloudWatch Metric Streams integration](/docs/infrastructure/amazon-integrations/get-started/introduction-aws-integrations/#monitor-aws) to get metrics, and [Forward AWS service logs to New Relic](/install/aws-logs/) to get logs into New Relic. This document describes how to get started with AWS monitoring and provides an overview about AWS costs. 
 
 ## Start with the CloudWatch Metric Streams integration [#monitor-aws]
 
-Our AWS observability solution is the Amazon CloudWatch Metric Streams integration, which collects data about your entire AWS stack with [Amazon CloudWatch Metric Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html). For the majority of use cases, you begin monitoring your AWS stack by [installing our Amazon CloudWatch Metric Streams integration](/install/aws-cloudwatch).
+The Amazon CloudWatch Metric Streams integration collects data about your entire AWS stack with [Amazon CloudWatch Metric Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html). For the majority of use cases, you begin monitoring your AWS stack by [installing our Amazon CloudWatch Metric Streams integration](/install/aws-cloudwatch).
 
 Our CloudWatch Metric Streams integration provides the best data about your AWS stack, but if you use [integrations not supported by CloudWatch Metric Streams](/docs/infrastructure/amazon-integrations/connect/connect-aws-new-relic-infrastructure-monitoring/#integrations-not-replaced-streams), you may need to use our legacy API polling solution. If you're currently using API polling for supported integrations, we recommend you [migrate your data to Amazon CloudWatch Metric Streams](/install/aws-cloudwatch/?task=migrate).
 

src/content/docs/logs/troubleshooting/aws-log-integration-issues/custom-attributes-not-appearing.mdx

@@ -0,0 +1,22 @@
+---
+title: Custom attributes not appearing in logs
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the custom attributes not appearing in the logs.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+Logs are not appearing in New Relic for a CloudWatch via Firehose integration setup.
+
+## Solution
+
+Possible reasons and their solutions:
+
+* For S3 via Lambda or CloudWatch via Lambda integrations, the `CUSTOM_META_DATA` environment variable may not be set, or it may not have a valid JSON value. Verify it.
+* For CloudWatch via Firehose, the key-value pairs may not be set correctly. Ensure your key-value pairs are present under <DNT>**Parameters**</DNT>.
+* Your key names may overlap with forbidden keywords.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/log-forwarding-restrictions.mdx

@@ -0,0 +1,24 @@
+---
+title: AWS Log forwarding restrictions
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Log forwarding restrictions
+metaDescription: 'Restrictions for log forwarding with New Relic.'
+freshnessValidatedDate: never
+---
+
+
+
+New Relic has the following restrictions with AWS integration for log forwarding when ingesting logs via S3 or Firehose:
+
+### For ingesting AWS logs from S3 [#s3-restrictions]
+* The log file size should be less that 80 MB.
+* The size of a single log line should be less than 8 MB.
+* Log lines between 1mb and 8mb will be split into multiple messages and forwarded to New Relic.
+* New Relic currently doesn't support Client Json or log line parsing.
+* New Relic currently supports only `Gzip` and `Bzip2` compression formats for log files. It processes files in other compressed formats as uncompressed files.
+### For ingesting logs from CloudWatch [#cloudwatch-restrictions]
+* New Relic currently doesn't support client Json or log line parsing.
+* New Relic currently supports only [Lambda layers](https://github.com/newrelic/newrelic-lambda-layers) for parsing logic.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-cloudwatch-firehose.mdx

@@ -0,0 +1,22 @@
+---
+title: Logs not appearing from CloudWatch via Firehose
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the Logs are not appearing from CloudWatch via Firehose.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+Logs are not appearing in New Relic for a CloudWatch via Firehose integration setup.
+
+## Solution
+
+Possible reasons and their solutions:
+
+* The API key may not be correct. Navigate to <DNT>**Amazon Data Firehose > Firehose streams > your-stream > Edit destination settings**</DNT>, and update the API key with the correct ingest license key.
+* The provided endpoint may be wrong. Verify it.
+* Check the S3 logs or Cloudwatch error logs.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-cloudwatch-lambda.mdx

@@ -0,0 +1,26 @@
+---
+title: Logs not appearing from CloudWatch via Lambda
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the Logs are not appearing from CloudWatch via Lambda.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+Logs are not appearing in New Relic for a CloudWatch via Lambda integration setup.
+
+## Solution
+
+Possible reasons and their solutions:
+
+* The New Relic ingest license key may not be valid. In your AWS account, find the resource with Logical Id `NewRelicLogsLicenseKeySecret` from <DNT>**CloudFormation stack > Resources**</DNT>, and verify the key.
+* To debug further, navigate to <DNT>**CloudFormation Stack > Resources > NewRelicServerlessLogForwarder (Lambda Function) > Configuration > Environment Variables**</DNT> and set the `DEBUG_MODE` parameter to true. Then, check the CloudWatch logs for the Lambda function to get the debug logs.
+* The event may have been a genuine failure. To check, navigate to <DNT>**CloudFormation Stack > Resources > NewRelicLogForwarderDLQ**</DNT>, or search for your DLQ in AWS SQS, and examine the DLQ resource.
+* The New Relic region and the ingest license key provided to the Lambda function or Firehose may not match. Verify and match it.
+
+  For example, if the New Relic region is EU, provide a New Relic ingest license key from the EU region.
+* The `NewRelicServerlessLogForwarder` may not have the CloudWatch trigger. Examine the output of the nested CloudFormation stacks.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-s3-lambda.mdx

@@ -0,0 +1,22 @@
+---
+title: Logs not appearing from S3 via Lambda
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the Logs are not appearing from S3 via Lambda.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+Logs are not appearing in New Relic for a S3 via Lambda integration setup.
+
+## Solution
+
+Possible reasons and their solutions:
+
+* Verify the possible reasons in [Logs not appearing from CloudWatch vai Lambda](/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-cloudwatch-lambda).
+* S3 files may not adhere to the [Log forwarding restrictions](/docs/logs/troubleshooting/aws-log-integration-issues/log-forwarding-restrictions) in New Relic.
+* The `NewRelicServerlessLogForwarder` may not have the S3 trigger. Check the output of the nested CloudFormation stacks, and add it.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/stack-creation-fails.mdx

@@ -0,0 +1,28 @@
+---
+title: Failing to create a Stack
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the stack creation fails.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+When creating a CloudFormation stack, it may fail due to various reasons.
+
+## Solution
+
+Possible reasons and their solutions:
+
+* The IAM role you have selected during the CloudFormation stack creation may not have sufficient [permissions](https://docs.aws.amazon.com/ARG/latest/userguide/gettingstarted-prereqs-permissions.html).
+* If you have not select an IAM role, make sure your account not have the required [permissions](/docs/accounts/accounts-billing/new-relic-one-user-management/user-permissions/#logs).
+* If the failure message appears as following, delete or rename the existing resources with that name, and retry.
+
+  `Resource type {resource-name} with identifier {resource-identifier-name} already exists.`
+
+* Check the output of the nested CloudFormation stacks to identify potential errors.
+* Ensure the selected S3 buckets don’t have event notifications on any object creation events.
+* Ensure the selected CloudWatch log groups don’t have similar subscription filters.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/stack-deletion-fails.mdx

@@ -0,0 +1,20 @@
+---
+title: Failing to delete a Stack
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the stack deletion fails.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+When deleting a CloudFormation stack, it may fail due to various reasons.
+
+## Solution
+
+Possible reason and its solution:
+
+* If the S3 buckets created by the stack (e.g., for Firehose error logs) is not empty, then it cannot be deleted. archive the contents and then retry deleting the stack.

src/install/aws-logs/aws_services/Install-aws-integration.mdx

@@ -0,0 +1,19 @@
+---
+headingText: Install the AWS integration for Logs forwarding
+componentType: default
+---
+
+Forward your AWS logs into New Relic to gain enhanced visibility into system performance, enable early detection of issues, and make informed decisions. Our log management solution improves your team's operational efficiency while maintaining security and compliance.
+
+
+Follow these steps to install the New Relic AWS integration and set up log forwarding from your AWS services.
+
+### Prerequisites [#prerequisites]
+Before you begin, ensure you have the following:
+* An AWS account with [permissions](https://docs.aws.amazon.com/ARG/latest/userguide/gettingstarted-prereqs-permissions.html) to deploy new AWS resources and [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)
+* A New Relic account with [permissions](/docs/accounts/accounts-billing/new-relic-one-user-management/user-permissions/#logs) to set up log forwarding
+* A New Relic [license key](https://one.newrelic.com/api-keys)
+
+<Callout variant="important">
+If you are already using Lambda for S3 or CloudWatch, delete it or remove triggers from it manually.
+</Callout>

src/install/aws-logs/aws_services/configure-log-forwarding.mdx

@@ -0,0 +1,42 @@
+---
+headingText: Configure logs forwarding
+componentType: default
+---
+
+In the <DNT>**Configuration logs**</DNT> page, choose a path for sending logs, set up integration for AWS sinks, verify the log endpoint, and add custom attributes to organize your logs.
+
+<Steps>
+<Step>
+### Choose your log forwarding path [#choose-log-forwarding-path]
+Following are the paths to forward AWS service logs to New Relic, suited to different needs.
+* <DNT>**Amazon Kinesis Data Firehose**</DNT>: Ideal for direct and high-volume data streaming.
+* <DNT>**New Relic Lambda Function**</DNT>: Provides high control over data transformation before forwarding.
+
+Depending upon your needs, choose log interation path(s):
+   * <DNT>**S3 via Lambda**</DNT>
+   * <DNT>**CloudWatch via Firehose or Lambda > CloudWatch via Firehose**</DNT>
+   * <DNT>**CloudWatch via Firehose or Lambda > CloudWatch via Lambda**</DNT>
+
+You can select both S3 via Lambda, and any one CloudWatch integration method together.
+</Step>
+<Step>
+### Choose AWS sink [#choose-aws-sink]
+You can use a CloudFormation template to set up log forwarding to New Relic for up to 40 buckets and/or log groups.
+1. To send AWS service logs stored in S3 buckets using the New Relic Lambda function, specify the following details:
+   * **S3 bucket name**: S3 source bucket names
+   * **Prefix** (Optional) : Specify a source prefix within the S3 buckets
+2. To send AWS service logs from CloudWatch log groups using the New Relic Lambda function or Firehose, specify the following details:
+   * **Log Group**: Source CloudWatch Log group names
+   * **Filter** (Optional): Specify a filter to refine logs within the CloudWatch Logs
+</Step>
+<Step>
+### Set API endpoint for logs [#choose-api-endpoint]
+The system sets the endpoint by default based on the region of your New Relic account.
+* US account: `https://log-api.newrelic.com/log/v1`
+* EU account: `https://log-api.eu.newrelic.com/log/v1`
+</Step>
+<Step>
+### Add custom attributes [#add-custom-attributes]
+Add custom attributes to organize your logs and make it easier to search, filter, analyze, and parse. After adding custom configurations, select <DNT>**Continue**</DNT>.
+</Step>
+</Steps>

src/install/aws-logs/aws_services/deploy-cloudformation-stack.mdx

@@ -0,0 +1,6 @@
+---
+headingText: Deploy CloudFormation stack
+componentType: default
+---
+
+With a single deployment command from New Relic, a stack generates in AWS that groups log forwarding from multiple AWS resources. Multiple stacks may be generated depending on how you configure log and metric forwarding.

src/install/aws-logs/aws_services/generate-cloudformation-template.mdx

@@ -0,0 +1,17 @@
+---
+headingText: Generate CloudFormation template
+componentType: default
+---
+
+Generate a CloudDormation template that has all the configuration details you have entered so far.
+
+1. Provide a unique name for your CloudFormation stack. The default name is `NewRelic-AWS-Integration`.
+2. Copy your existing license key. If you do not have a key, select <DNT>**Create a new key**</DNT> and copy it.
+3. Select <DNT>**Launch your Cloudformation in AWS**</DNT>. AWS Cloud Console opens with the <DNT>**Quick create stack**</DNT> form.
+4. The template includes all the parameters you have entered in New Relic. The following are the details you need to fill in:
+   * <DNT>**New Relic Ingest License Key**</DNT>: Paste the copied license key.
+   * <DNT>**Store New Relic License Key in AWS Secrets Manager**</DNT>: If you want to store the New Relic license key in AWS Secrets Manager as part of the Lambda environment variable, keep it **true** (default value).
+   * (Optional) In the <DNT>**Permissions**</DNT> block, select the IAM role for CloudFormation to use for all stack operations.
+   * Read through <DNT>**Launch your Cloudformation in AWS**</DNT> and acknowledge the necessary capabilities.
+5. Select <DNT>**Create stack**</DNT>. It may take several minutes to create the new stack.
+If it fails to create a stack, refer the [troubleshooing procedures](/docs/logs/troubleshooting/aws-log-integration-issues/stack-creation-fails).

src/install/aws-logs/aws_services/see-your-data.mdx

@@ -0,0 +1,37 @@
+---
+headingText: See your data
+componentType: default
+---
+To complete the AWS account integration in New Relic, follow these steps:
+1. Go back to the <DNT>**Connect your AWS Account**</DNT> page in the New Relic platform instance.
+2. Provide the licence key:
+   * If you are using an existing license key, paste it in <DNT>**License key**</DNT> and continue.
+   * If you have created a new key, just select <DNT>**Continue**</DNT>. 
+9. Select <DNT>**See your data**</DNT>.
+
+If everything is configured correctly and your data is being collected, you should see log data in both of these places:
+
+* Our [logs UI](https://one.newrelic.com/launcher/logger.log-launcher)
+* New Relic tools for running [NRQL queries](/docs/chart-builder/use-chart-builder/choose-data/use-advanced-nrql-mode-specify-data).
+For example, you can run a query like this:
+
+   S3 lambda / CloudWatch lambda:
+   ```
+   SELECT * FROM Log WHERE instrumentation.provider = 'aws' AND instrumentation.name = 'lambda'
+   ```
+   Cloudwatch firehose:
+   ```
+   SELECT * FROM Log WHERE instrumentation.provider = 'aws' AND instrumentation.name = 'firehose'
+   ```
+
+If no data appears after you enable the log management capabilities, or the custom attributes are not appearing in the log, refer to the following troubleshooing procedures:
+* [Logs not appearing from CloudWatch via Firehose](/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-cloudwatch-firehose)
+* [Logs not appearing from CloudWatch via Lambda](/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-cloudwatch-lambda)
+* [Logs not appearing from S3 via Lambda](/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-s3-lambda)
+* [Custom attributes not appearing in logs](/docs/logs/troubleshooting/aws-log-integration-issues/custom-attributes-not-appearing)
+* [Failing to delete a Stack](/docs/logs/troubleshooting/aws-log-integration-issues/stack-deletion-fails)
+* [Log forwarding restrictions](/docs/logs/troubleshooting/aws-log-integration-issues/log-forwarding-restrictions)
+
+### Disable log forwarding [#disable]
+
+To disable log forwarding capabilities, follow standard procedures in [Amazon CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html). You do not need to do anything else in New Relic.

src/install/aws-logs/aws_services/set-up-aws-integration-with-log-forwarding.mdx

@@ -0,0 +1,15 @@
+---
+headingText: Set up your AWS integration with log forwarding
+componentType: default
+---
+
+Set up an AWS integration with a CloudFormation template via New Relic and specify the data type you want to send to New Relic. You can set up the integration for forwarding both logs and metrics using a single CloudFormation template.
+1. Log in to [New Relic](https://one.newrelic.com/).
+2. Navigate to <DNT>**+ Integrations & Agents**</DNT>.
+3. Search and select <DNT>**Integrate your AWS account**</DNT> from the <DNT>**All**</DNT> tab.
+4. From the <DNT>**Select an account**</DNT> page, select your New Relic account and continue.
+5. From the <DNT>**Choose data type(s)**</DNT> page, select <DNT>**Logs**</DNT> and continue.
+<Callout variant="tip">
+To set up integration for forwarding metrics data, refer [Amazon CloudWatch Metric Streams](/install/aws-cloudwatch/).
+</Callout>
+6. From the <DNT>**Choose a setup method**</DNT> page, Select <DNT>**Automate AWS with CloudFoundation**</DNT> and continue. This is the only setup method available for log integration.