diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 202f0a550b1de..81e91f9aa4e5b 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -2320,6 +2320,11 @@ menu: parent: platform_heading identifier: change_tracking weight: 140000 + - name: Feature Flags + url: change_tracking/feature_flags/ + parent: change_tracking + identifier: change_tracking_feature_flags + weight: 1 - name: Service Level Objectives url: service_management/service_level_objectives/ pre: slos diff --git a/content/en/change_tracking/_index.md b/content/en/change_tracking/_index.md index 4d8ad26ebe349..788fb54104d16 100644 --- a/content/en/change_tracking/_index.md +++ b/content/en/change_tracking/_index.md @@ -45,7 +45,7 @@ Change Tracking streamlines troubleshooting and incident response by surfacing r Change Tracking supports monitoring of a range of modifications to your service and its dependencies including: - Deployments -- Feature Flags +- [Feature Flags][14] - Configuration Changes - Database Modifications - Schema Changes @@ -120,49 +120,21 @@ To see relevant changes within the timeline and as overlays on your dashboard, e ## Tracked changes Change Tracking follows these types of changes across your infrastructure: -| Change Type | Tracking Requirements | -|-----------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------| -| Code Deployments (APM) | APM & [Deployment Tracking][4]. A version must be available on the service. | -| Kubernetes Deployment Manifest Updates | Datadog Agent Set Up for Kubernetes (Add service label to kubernetes yaml file if possible) | -| LaunchDarkly Feature Flag Events (service tag must be defined on event) | Third Party Datadog Integrations ([LaunchDarkly only][5]) | -| Custom Feature Flag Events | [Event Management API][6] | -| Custom Configuration Change Events | [Event Management API][6] | -| Watchdog Alerts (Error Rate Spikes, Latency Spikes, Cloud and API Outages, etc.) | See [Watchdog][7] documentation to learn more about requirements for specific Watchdog Alerts. | -| CrashLoopBackOff Kubernetes Pod Crashes | Kubernetes Integration (Add service label to kubernetes yaml file if possible) | -| PostgreSQL, SQL Server and MySQL Database Table (Schemas) Change | See [Exploring Database Schemas][12] documentation to learn more about tracking schemas using DBM, and [Correlate Database Monitoring and Traces][10] to set up APM and DBM correlation | -| MongoDB Index & SearchIndex Changes | [Database Monitoring (DBM)][8], [Correlate Database Monitoring and Traces][10] | -| PostgreSQL Database Settings Change | [Database Monitoring (DBM)][8], [Correlate Database Monitoring and Traces][10] | -| SQL Server Database Settings Change | [Database Monitoring (DBM)][8], [Correlate Database Monitoring and Traces][10] | -| Kafka Schema Updates | [Data Streams Monitoring (DSM)][9] | -| Manual Kubernetes Deployment Scale Events | Kubernetes Audit Logging | -| Cloud Infrastructure Resource Changes (AWS, Azure, GCP) | [Resource Changes][13] - Enable Resource Collection and optionally cloud provider event forwarding | - -### Optional enrichment for feature flag changes -Change Tracking offers an optional way to enhance visibility into feature flag changes by automatically detecting affected services when tracing is set up for the feature flag client. This enhancement enables faster and more precise root cause analysis, especially when feature flag changes impact multiple services. - -#### To enable auto-enrichment of feature flag changes - -Auto-enrichment helps you identify which services are affected by feature flag changes by automatically detecting service dependencies through tracing. To set this up: - -1. Add tracing around your feature flag client code. -2. Name the trace operation **experiments.IsEnabled**. -3. Add a tag called **experiment_id**. Set its value to match the ID of the relevant feature flag. - -#### To toggle LaunchDarkly feature flags from the details panel - -If you discover that a feature flag change is causing issues, you can remediate it by toggling the flag directly from the Change Tracking side panel. - -
This feature is powered by Workflow Automation and is subject to Workflow Automation pricing.
- -To enable and use this feature: - -1. Set up workflow connections following the [Workflow Connections documentation][11]. -1. Navigate to the service page you want to investigate. -1. Locate the changes timeline in the **Service Summary** section. -1. Click any feature flag change in the Change Tracking timeline to open the details panel. - {{< img src="/change_tracking/feature-flag-toggle.png" alt="Click feature flag events to view more detail and toggle the feature flag." style="width:90%;" >}} - -1. Click **Toggle Feature Flag** to turn the feature flag on or off. +| Change Type | Tracking Requirements | +|----------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Code Deployments (APM) | APM & [Deployment Tracking][4]. A version must be available on the service. | +| Kubernetes Deployment Manifest Updates | Datadog Agent Set Up for Kubernetes (Add service label to Kubernetes yaml file if possible) | +| Feature Flags | Use the LaunchDarkly integration or send custom events using the Events API. See the [Feature Flag Tracking documentation][14] for setup and advanced options. | +| Custom Configuration Change Events | [Event Management API][6] | +| Watchdog Alerts (Error Rate Spikes, Latency Spikes, Cloud and API Outages, etc.) | See [Watchdog][7] documentation to learn more about requirements for specific Watchdog Alerts. | +| CrashLoopBackOff Kubernetes Pod Crashes | Kubernetes Integration (Add service label to Kubernetes yaml file if possible) | +| PostgreSQL, SQL Server and MySQL Database Table (Schemas) Change | See [Exploring Database Schemas][12] documentation to learn more about tracking schemas using DBM, and [Correlate Database Monitoring and Traces][10] to set up APM and DBM correlation | +| MongoDB Index & SearchIndex Changes | [Database Monitoring (DBM)][8], [Correlate Database Monitoring and Traces][10] | +| PostgreSQL Database Settings Change | [Database Monitoring (DBM)][8], [Correlate Database Monitoring and Traces][10] | +| SQL Server Database Settings Change | [Database Monitoring (DBM)][8], [Correlate Database Monitoring and Traces][10] | +| Kafka Schema Updates | [Data Streams Monitoring (DSM)][9] | +| Manual Kubernetes Deployment Scale Events | Kubernetes Audit Logging | +| Cloud Infrastructure Resource Changes ({{< tooltip text="in Preview" tooltip="This feature is in Preview and currently limited to a small sample of cloud resource changes. To request access, see the Resource Changes documentation linked in Tracking Requirements." >}}) | [Resource Changes][13] - Enable Resource Collection and optionally cloud provider event forwarding ## Further reading @@ -180,4 +152,5 @@ To enable and use this feature: [10]: /database_monitoring/connect_dbm_and_apm/ [11]: /service_management/workflows/connections/#work-with-connections [12]: /database_monitoring/schema_explorer -[13]: /infrastructure/resource_catalog/resource_changes/ \ No newline at end of file +[13]: /infrastructure/resource_catalog/resource_changes/ +[14]: /change_tracking/feature_flags \ No newline at end of file diff --git a/content/en/change_tracking/feature_flags.md b/content/en/change_tracking/feature_flags.md new file mode 100644 index 0000000000000..f38162e5413bc --- /dev/null +++ b/content/en/change_tracking/feature_flags.md @@ -0,0 +1,160 @@ +--- +title: Track Feature Flag Changes +further_reading: + - link: "/change_tracking/" + tag: "Documentation" + text: "Change Tracking" + - link: "/integrations/launchdarkly/#feature-flag-tracking-integration/" + tag: "Documentation" + text: "LaunchDarkly" +--- + +## Overview + +Feature flag tracking in Datadog helps you correlate flag changes with system performance issues, accelerating incident resolution. By tracking when flags are toggled, updated, or rolled out, you can identify whether a feature flag change caused a performance degradation or outage. + +With feature flag tracking, you can: +- View flag changes directly in your dashboards, monitors, and service pages +- Automatically identify which services are affected by a flag change +- Roll back problematic flags directly from inside Datadog + +{{< img src="/change_tracking/monitor.png" alt="The monitor graph shows the error spike correlated with the flag change." style="width:100%;" >}} + +## Track feature flags + +Datadog supports tracking LaunchDarkly flags using the [LaunchDarkly integration][1] or tracking feature flags from other feature flag providers using the [Events API][3]. + +### LaunchDarkly flags + +To track LaunchDarkly feature flags in your services' Change Tracking timeline: + +1. Enable the [LaunchDarkly integration][1] in Datadog. +1. Go to **Flags > `` > Settings** in LaunchDarkly. +1. In **Custom properties**, add a tag with key `service` and value ``, matching your Datadog service name exactly. +1. Click **Save changes**. + +For example, to link a flag to the `payments_api` service used in the examples below, you would set the tag value to `payments_api`. After you submit the event, you can navigate to the [Software Catalog][7], select the `payments_api` service, and see the `fallback_payments_test` feature flag event in the Change Tracking timeline. + +### Custom feature flags + +Send feature flag events from any provider using the [Events API][3]. Create a `change` category event and include a service tag to link the event to your service. + +Example request: + +```json +{ + "data": { + "attributes": { + "aggregation_key": "string", + "attributes": { + "author": { + "name": "datadog@datadog.com", + "type": "user" + }, + "change_metadata": { + "dd": { + "team": "datadog_team", + "user_email": "datadog@datadog.com", + "user_id": "datadog_user_id", + "user_name": "datadog_username" + }, + "resource_link": "datadog.com/feature/fallback_payments_test" + }, + "changed_resource": { + "name": "fallback_payments_test", + "type": "feature_flag" + }, + "impacted_resources": [ + { + "name": "payments_api", + "type": "service" + } + ], + "new_value": { + "enabled": true, + "percentage": "50%", + "rule": { + "datacenter": "devcycle.us1.prod" + } + }, + "prev_value": { + "enabled": true, + "percentage": "10%", + "rule": { + "datacenter": "devcycle.us1.prod" + } + } + }, + "category": "change", + "message": "payment_processed feature flag has been enabled", + "tags": [ + "env:test" + ], + "timestamp": "string", + "title": "payment_processed feature flag updated" + }, + "type": "event" + } +} +``` + +## Automatically detect affected services + +In addition to tracking when a feature flag's configuration changes using the LaunchDarkly integration or the Events API, you can also automatically identify every service that evaluates a flag. This auto-enrichment provides deeper context by using APM traces to show the real-time impact of a flag, which is especially useful when a single flag is used by multiple services. + +### Setup + +To automatically detect services using a feature flag, instrument your feature flag evaluation code with the APM tracing library. This allows Datadog to automatically detect all services that evaluate a specific flag, even if they weren't originally tagged. + +1. [Instrument your feature flag evaluation code][4] using the Datadog tracing library. +1. Create a custom span with the operation name `experiments.IsEnabled` to track feature flag evaluations. +3. Tag the span with `experiment_id:`, where `` matches the feature flag ID. + +For example: + +```python +# Trace feature flag evaluation to enable auto-detection +with tracer.trace("experiments.IsEnabled") as span: + span.set_tag("experiment_id", "fallback_payments_test") + # Your existing feature flag evaluation code + flag_value = evaluate_flag("fallback_payments_test") +``` + +## Remediate feature flag changes with Workflow Automation + +When you identify that a feature flag change caused an issue, you can immediately toggle its state without leaving Datadog. This feature uses [Workflow Automation][2] to toggle LaunchDarkly flags directly from Change Tracking timelines. + +
This feature requires Workflow Automation. See Workflow Automation pricing.
+ +### Setup + +To set up feature flag toggles using Workflow Automation: + +1. Go to [**Actions > Action Catalog > Connections**][6]. +1. Click **New Connection**. +1. Choose *LaunchDarkly*. +1. Complete the required information, then click **Next, Confirm Access**. +1. Set access permissions for the connection. +1. Click **Create**. + +### Use feature flag toggles + +To toggle feature flags on or off from inside Datadog: + +1. Click a LaunchDarkly feature flag change in the Change Tracking timeline. +1. Click the **Toggle Feature Flag** button. +1. Click **Run Action** to run the workflow and toggle the feature flag on or off. + +{{< img src="/change_tracking/toggle.png" alt="The details panel for a LaunchDarkly feature flag event, showing the 'Toggle Feature Flag' button." style="width:90%;" >}} + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /integrations/launchdarkly/#setup +[2]: https://app.datadoghq.com/actions/connections +[3]: /api/latest/events/ +[4]: /tracing/trace_collection/ +[5]: https://app.datadoghq.com/apm/settings +[6]: https://app.datadoghq.com/actions/connections +[7]: https://app.datadoghq.com/software diff --git a/static/images/change_tracking/connection.png b/static/images/change_tracking/connection.png new file mode 100644 index 0000000000000..bbb160f5a6476 Binary files /dev/null and b/static/images/change_tracking/connection.png differ diff --git a/static/images/change_tracking/monitor.png b/static/images/change_tracking/monitor.png new file mode 100644 index 0000000000000..417801a08fbf2 Binary files /dev/null and b/static/images/change_tracking/monitor.png differ diff --git a/static/images/change_tracking/toggle.png b/static/images/change_tracking/toggle.png new file mode 100644 index 0000000000000..e937494faf05d Binary files /dev/null and b/static/images/change_tracking/toggle.png differ