Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: Fix README links and formatting #296

Merged
merged 2 commits into from
Oct 2, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions logs/fluent-bit/ecs-fargate/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The aws-for-fluent-bit image, [maintained here by AWS](https://github.com/aws/aw

The base_filters.conf file includes a set of filters to ensure proper ingestion by our backend. This should be included as the first configuration file for your instance deployment. Ensure you upload this to an S3 bucket in your AWS account.

As I just alluded to, you can load multiple configuration files from S3 to build your final configuration. This is done by setting custom environment variables within the task definition.
As I just alluded to, you can load multiple configuration files from S3 to build your final configuration. This is done by setting custom environment variables within the task definition.

Example container declaration within a Task Definition:

Expand Down Expand Up @@ -71,6 +71,7 @@ In order to allow container access to the S3 object, you'll need to provide the
]
}
```

Note: Don't confuse Task Execution Role for Task Role, this permission needs to be added to the Task Role. (Contrary to the ADOT (OTEL) Metrics and Traces integration)

After you've added the above container to your existing Task Definition, you need to adjust the logConfiguration for the containers you wish to forward to Coralogix.
Expand All @@ -92,4 +93,4 @@ To do so, you'd add this "logConfiguration" section to each of your application
"compress": "gzip"
}
}
```
```
2 changes: 1 addition & 1 deletion logs/fluent-bit/eks-fargate/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@

### v0.0.1 / 2023-08-17

* [DOC] Documented EKS Fargate fluent-bit (log_router) integration
* [DOC] Documented EKS Fargate fluent-bit (log_router) integration
6 changes: 3 additions & 3 deletions logs/fluent-bit/eks-fargate/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,13 +97,13 @@ data:
Time_Keep true
```

Details on the workings of the AWS log router can be found in the configuration documentation on the AWS docs site here:
Details on the workings of the AWS log router can be found in the configuration documentation on the AWS docs site here:

[Fargate logging - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html)

## Notes:

- The log_router will only be attached to workloads started after the manifest has been applied. **You will need to restart your pods** in order for the log_router to start forwarding the logs to your Firehose.
- You can add additional filters, but they are limited to the following types:
`grep, parser, record_modifier, rewrite_tag, throttle, nest, modify, kubernetes`
- If you wish to add a Cloudwatch output, you’ll have to add additional permissions to your Fargate profile. Please review the above linked AWS documentation for details.
`grep, parser, record_modifier, rewrite_tag, throttle, nest, modify, kubernetes`
- If you wish to add a Cloudwatch output, you’ll have to add additional permissions to your Fargate profile. Please review the above linked AWS documentation for details.
1 change: 0 additions & 1 deletion otel-agent/k8s-helm/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
* [FEATURE] Set integrationName as resource attribute
* [CHORE] Upgrading upstream chart. (v0.71.2)


### v0.0.33 / 2023-09-04

* [CHORE] Upgrading upstream chart. (v0.71.1)
Expand Down
15 changes: 12 additions & 3 deletions otel-ecs-fargate/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

The OpenTelemetry collector offers a vendor-agnostic implementation of how to receive, process and export telemetry data.

In this document, we'll explain how to add the OTEL collector as a sidecar agent to your ECS Task Definitions. We use an AWS customized OpenTelemetry image called AWS Distro for OpenTelemetry (ADOT), as it has several features that allow for more convenient management of the configuration. We also have an example cloudformation template for review [here](https://github.com/coralogix/cloudformation-coralogix-aws/tree/master/aws-integrations/ecs-fargate)
In this document, we'll explain how to add the OTEL collector as a sidecar agent to your ECS Task Definitions. We use an AWS customized OpenTelemetry image called AWS Distro for OpenTelemetry (ADOT), as it has several features that allow for more convenient management of the configuration. We also have an example cloudformation template for review [here](https://github.com/coralogix/cloudformation-coralogix-aws/tree/master/aws-integrations/ecs-fargate)

The ADOT image, [maintained here by AWS](https://github.com/aws-observability/aws-otel-collector), allows for loading of the OpenTelemetry configuration via Systems Manager Parameter Stores. This makes adjusting your configuration more convenient and more dynamic than baking a static configuration into your container image.

Expand Down Expand Up @@ -74,7 +74,15 @@ Example container declaration within a Task Definition:
]
```

In the example above, you'll need to set two instances each of <Coralogix PrivateKey> and <Coralogix Domain>. The logConfiguration section included in the example will forward OTEL logs to the Coralogix platform, as documented in our fluentbit log processing configuration instructions [here](../../logs/fluent-bit/ecs-fargate/README.md). If you don't want to have them submitted to the Coralogix platform, you can replace the logConfiguration with whichever logDriver configuration you would prefer. To submit to Cloudwatch, you can configure as so:
In the example above, you'll need to set two instances each of

<coralogix privatekey="">

and

<coralogix domain="">

. The logConfiguration section included in the example will forward OTEL logs to the Coralogix platform, as documented in our fluentbit log processing configuration instructions [here](../logs/fluent-bit/ecs-fargate/README.md). If you don't want to have them submitted to the Coralogix platform, you can replace the logConfiguration with whichever logDriver configuration you would prefer. To submit to Cloudwatch, you can configure as so:

```
"logConfiguration": {
Expand Down Expand Up @@ -106,6 +114,7 @@ In order to allow container access to the Systems Manager Parameter Store, you'l
]
}
```

Note: Don't confuse Task Role for Task Execution Role, this permission needs to be added to the Task Execution Role. (Contrary to the fluentbit Logs integration)

After adding the above container to your existing Task Definition, your applications can submit their traces and metrics exports to http://localhost:4318/v1/traces and /v1/metrics. It will also collect container metrics from all containers in the Task Definition.
After adding the above container to your existing Task Definition, your applications can submit their traces and metrics exports to http://localhost:4318/v1/traces and /v1/metrics. It will also collect container metrics from all containers in the Task Definition.
2 changes: 1 addition & 1 deletion otel-eks-fargate/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
### v0.0.1 / 2023-08-17

* [NEW] Added EKS Fargate related configuration files
* [DOC] Documented EKS Fargate OTEL integration
* [DOC] Documented EKS Fargate OTEL integration
14 changes: 7 additions & 7 deletions otel-eks-fargate/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,19 @@ The Coralogix EKS Fargate integration for Metrics and Traces leverages OpenTelem
## Requirements:

- `cx-eks-fargate-otel` namespace declared in your EKS cluster
- This namespace need not be hosted by a Fargate profile, but if it is desired, you’ll need to create one.
- This namespace need not be hosted by a Fargate profile, but if it is desired, you’ll need to create one.
- A Secret containing your Coralogix API Key, in the `cx-eks-fargate-otel` namespace.

## Creating Secret

1. Export your API key to a local variable:
1. `export PRIVATE_KEY=<Send-Your-Data API key>`
1. `export PRIVATE_KEY=<Send-Your-Data API key>`
2. Set your namespace variable:
1. `export NAMESPACE=cx-eks-fargate-otel`
1. `export NAMESPACE=cx-eks-fargate-otel`
3. Create the secret using kubectl:
1. `kubectl create secret generic coralogix-keys -n $NAMESPACE --from-literal=PRIVATE_KEY=$PRIVATE_KEY`
1. `kubectl create secret generic coralogix-keys -n $NAMESPACE --from-literal=PRIVATE_KEY=$PRIVATE_KEY`
4. Confirm it’s been set
1. `kubectl get secret coralogix-keys -o yaml -n $NAMESPACE`
1. `kubectl get secret coralogix-keys -o yaml -n $NAMESPACE`

## Create ServiceAccount

Expand Down Expand Up @@ -52,7 +52,7 @@ eksctl create iamserviceaccount \

## Configure and Deploy OTEL Collector Service:

The attached yaml manifest will deploy an OTEL collector, a clusterIP service for submission of application traces and metrics, and the cluster permissions required to query the Kubernetes API.
The attached yaml manifest will deploy an OTEL collector, a clusterIP service for submission of application traces and metrics, and the cluster permissions required to query the Kubernetes API.

There are a few container environment variables that need to be set, detailed at the top of the yaml file.

Expand All @@ -76,4 +76,4 @@ This manifest also has some required environmental variables that need to be set

Again, after setting the environment variables, deploy using kubectl apply:

`kubectl apply -f cx-eks-fargate-otel-self-monitoring.yaml`
`kubectl apply -f cx-eks-fargate-otel-self-monitoring.yaml`