Merge pull request #9682 from DataDog/pcarioufr/unified_log_explorer

Document the unified Query Editor for the Log Explorer

Author: sarina-dd

config/_default/menus/menus.en.yaml

@@ -1694,30 +1694,18 @@ main:
     url: logs/explorer/live_tail/
     parent: log_explorer
     weight: 601
-  - name: Log List
-    url: logs/explorer/list/
+  - name: Log Side Panel
+    url: logs/explorer/side_panel/
     parent: log_explorer
     weight: 602
-  - name: Log Patterns
-    url: logs/explorer/patterns/
-    parent: log_explorer
-    weight: 603
-  - name: Log Transactions
-    url: logs/explorer/transactions/
-    parent: log_explorer
-    weight: 604
-  - name: Log Analytics
-    url: logs/explorer/analytics/
-    parent: log_explorer
-    weight: 605
   - name: Saved Views
     url: logs/explorer/saved_views/
     parent: log_explorer
-    weight: 606
+    weight: 603
   - name: Facets
     url: logs/explorer/facets/
     parent: log_explorer
-    weight: 607
+    weight: 604
   - name: Search Syntax
     url: logs/search_syntax/
     parent: log_management

content/en/logs/explorer/_index.md

@@ -1,87 +1,42 @@
 ---
 title: Live Tail
 kind: documentation
-description: 'See all your logs in real time.'
+description: 'Search through all of your logs and perform Log Analytics'
 aliases:
-    - /logs/live_tail
+    - /logs/explore/livetail
 further_reading:
-    - link: 'logs/explorer/analytics'
-      tag: 'Documentation'
-      text: 'Perform Log Analytics'
     - link: 'logs/processing'
       tag: 'Documentation'
       text: 'Learn how to process your logs'
-    - link: 'logs/processing/parsing'
+    - link: 'logs/explorer/side_panel'
+      tag: 'Documentation'
+      text: 'The log side panel'
+    - link: 'logs/explorer/#list-of-logs'
       tag: 'Documentation'
-      text: 'Learn more about parsing'
+      text: 'The list view of logs'
 ---
 
-{{< img src="logs/explorer/livetail/live_tail_demo.mp4" alt="Live tail" video="true"  >}}
-
 ## Overview
 
-The Live Tail feature gives you the ability to see all your log events in near real time from anywhere in your infrastructure. It displays logs as soon as they get out of the [Pipeline section][1] and before they are [indexed][2] by Datadog, hence:
-
-1. All logs ingested by Datadog are displayed. ([It's Logging without Limits][2]\*)
-2. Displayed logs have been processed.
-3. The stream can be paused.
-4. You can't go back in time.
-
-This feature allows you, for instance, to check if a process has correctly started, or if a new deployment went smoothly.
-
-## Live Tail view
-
-Choose the `Live Tail` option in the time range selector to switch to the Live Tail view:
-
-{{< img src="logs/explorer/livetail/live_tail_time_selector.png" alt="Live Tail time selector"  >}}
-
-The number of received events per second is displayed at the top left, as well as the sampling rate. Since a stream of thousands of logs per second is not human readable, high throughput log streams are sampled.
-
-Use the [Live Tail search bar filtering features](#filtering-the-log-stream) to filter the log stream and the **Pause/Play** button at the top right of the screen to pause or resume the stream.
-
-**Note**: Selecting any log pauses the stream and displays more details about the selected log.
-
-### Display Options
-
-Customize the Live Tail view to better highlight the relevant information in your logs.
-Click on the gear at the top right of the page to activate one of the options below:
-
-{{< img src="logs/explorer/livetail/live_tail_column.png" alt="Live tail column"  style="width:30%;">}}
-
-1. Choose to display one, three, or ten lines from your logs attributes in your logstream.
-2. Enable/Disable the Date and Message column.
-3. Add any log attribute as a column either in this panel or by clicking on it directly:
-
-{{< img src="logs/explorer/livetail/live_tail_add_as_column.png" alt="Live tail add as column"  style="width:50%;">}}
-
-## Filtering the log Stream
-
-A valid query in the search bar displays logs that match your search criteria.
-The search syntax is the same in the Live Tail views as in the other Log views, but here, your query is matched against all of the ingested logs and not just the indexed ones.
-
-### JSON attributes
-
-Any query that works in other views works in the Live Tail view, but you can even go further and **filter on attributes that are not defined as facets**.
-
-For example, to filter on the following `filename` attribute there are two options:
+With Live Tail, access all your log events in near real time from anywhere in your infrastructure. The Live Tail view provides visibility on **all** logs, whether you choose to index them or not - see also [Exclusion Filters][1] on logs indexes. Logs flowing through the Live Tail are all structured, processed, and enriched from [Log Pipelines][2].
 
-{{< img src="logs/explorer/livetail/live_tail_save.png" alt="Live tail save"  style="width:50%;">}}
+For example, Live Tail is specifically useful to check if a process has correctly started or if a new deployment went smoothly.
 
-1. Click on the attribute and add it to the search:
+## Live Tail View
 
-    {{< img src="logs/explorer/livetail/live_tail_click_attribute.png" alt="Live tail click attribute"  style="width:50%;">}}
+In the [Log Explorer][3], choose the Live Tail option in the timerange to query logs as they flow into Datadog.
 
-2. Use the following query `@filename:runner.go`:
+{{< img src="logs/explorer/live_tail/livetail.gif" alt="Log Live Tail" style="width:100%;" >}}
 
-    {{< img src="logs/explorer/livetail/live_tail_filtered.png" alt="Live tail filtered"  style="width:50%;">}}
+Contrary to queries on indexed logs happening in the [Log Explorer][3], queries in the Live Tail do *not* require that you [declare a facet][4] beforehand.
 
-To filter on all logs with a line number above 150 use the following query: `@linenumber:>150`
+**Note**: For the sake of readability, the Live Tail output is sampled when too many logs matching the query are flowing in. The sampling applied is uniformly random, so that your Live Tail logs are statistically representative of your actual log throughput. Scope your query down with additional search filters if you need visibility on every single log flowing in.
 
 ## Further Reading
 
 {{< partial name="whats-next/whats-next.html" >}}
-<br>
-\*Logging without Limits is a trademark of Datadog, Inc.
 
-[1]: /logs/processing/pipelines/
-[2]: /logs/
+[1]: /logs/indexes#exclusion-filters
+[2]: /logs/processing
+[3]: /logs/explorer
+[4]: /logs/explorer/facets/