fix(metrics): raise inter-token latency histogram ceiling to 80s (ai-dynamo#14839)

Signed-off-by: Jie Hao <jihao@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: jh-nv

docs/fern/pages/reference/components/frontend-configuration.mdx

@@ -539,29 +539,29 @@ See the [KServe gRPC Frontend](../../developer-guide/knowledge-base/modular-comp
 
 ### Histogram buckets
 
-The frontend's six latency and size histograms use log-spaced buckets, rounded to two significant figures, always starting with a `0` edge. Each histogram is tuned by three environment variables that share a prefix: `_MIN` and `_MAX` bound the range, and `_COUNT` is the total number of bucket edges **including** that leading `0`, so `COUNT - 1` edges are spread across the range. The inter-token latency default of `13` therefore means `0` plus twelve log-spaced edges. Every edge, including the top one, is rounded to two significant figures, so the highest exported edge is `MAX` rounded rather than `MAX` itself — see the `Top le` column below. These have no CLI equivalent, and they are read once while the frontend builds its metrics, so changing one requires a restart.
+The frontend's six latency and size histograms use log-spaced buckets, rounded to two significant figures, always starting with a `0` edge. Each histogram is tuned by three environment variables that share a prefix: `_MIN` and `_MAX` bound the range, and `_COUNT` is the total number of bucket edges **including** that leading `0`, so `COUNT - 1` edges are spread across the range. The inter-token latency default of `20` therefore means `0` plus nineteen log-spaced edges. Every edge, including the top one, is rounded to two significant figures, so the highest exported edge is `MAX` rounded rather than `MAX` itself — see the `Top le` column below. These have no CLI equivalent, and they are read once while the frontend builds its metrics, so changing one requires a restart.
 
 | Histogram | Variable prefix | Min | Max | Count | Top `le` |
 |---|---|---|---|---|---|
 | `dynamo_frontend_request_duration_seconds` | `DYN_METRICS_REQUEST_DURATION` | `1.0` | `512.0` | `10` | `510` |
 | `dynamo_frontend_input_sequence_tokens` | `DYN_METRICS_INPUT_SEQUENCE` | `50.0` | `128000.0` | `12` | `130000` |
 | `dynamo_frontend_output_sequence_tokens` | `DYN_METRICS_OUTPUT_SEQUENCE` | `50.0` | `32000.0` | `10` | `32000` |
 | `dynamo_frontend_time_to_first_token_seconds` | `DYN_METRICS_TTFT` | `0.001` | `480.0` | `18` | `480` |
-| `dynamo_frontend_inter_token_latency_seconds` | `DYN_METRICS_ITL` | `0.001` | `2.0` | `13` | `2` |
+| `dynamo_frontend_inter_token_latency_seconds` | `DYN_METRICS_ITL` | `0.001` | `80.0` | `20` | `80` |
 | `dynamo_frontend_embedding_latency_seconds` | `DYN_METRICS_EMBEDDING_LATENCY` | `0.001` | `10.0` | `14` | `10` |
 
-For example, to raise the inter-token latency ceiling from 2 seconds to 80:
+For example, to raise the output sequence ceiling from 32k tokens to 128k:
 
 ```bash
-export DYN_METRICS_ITL_MAX=80
-export DYN_METRICS_ITL_COUNT=20
+export DYN_METRICS_OUTPUT_SEQUENCE_MAX=131072
+export DYN_METRICS_OUTPUT_SEQUENCE_COUNT=12
 ```
 
 Rounding can move the top edge either way, so pick a `MAX` that is already two significant figures if you need an exact ceiling. Prometheus buckets are inclusive, so `MAX=123` exports `le=120` and only observations *greater than* 120 and up to 123 fall into `+Inf` — a value of exactly 120 still lands in the `le=120` bucket. `MAX=131072` exports `le=130000`. Two of the defaults above are affected: request duration is set to `512` but exports `510`, and input sequence is set to `128000` but exports `130000`.
 
 `dynamo_frontend_cached_tokens` shares the input sequence buckets, so `DYN_METRICS_INPUT_SEQUENCE_*` changes both histograms.
 
-A histogram's configuration is validated as a set: `MIN` must be above zero, below `MAX`, and `COUNT` must be between 1 and 512. If any part fails, the frontend warns and reverts **all three** values for that histogram together — setting `DYN_METRICS_ITL_MIN=5` against the default `MAX` of `2.0` also discards a perfectly valid `DYN_METRICS_ITL_COUNT=20`.
+A histogram's configuration is validated as a set: `MIN` must be above zero, below `MAX`, and `COUNT` must be between 1 and 512. If any part fails, the frontend warns and reverts **all three** values for that histogram together — setting `DYN_METRICS_ITL_MIN=100` against the default `MAX` of `80.0` also discards a perfectly valid `DYN_METRICS_ITL_COUNT=20`.
 
 A value that cannot be parsed at all is handled per variable, not as a set: it warns, that one variable falls back to its default, and the others still apply.
 

lib/llm/src/http/service/metrics.rs

@@ -827,7 +827,7 @@ impl Metrics {
     /// - `DYN_METRICS_INPUT_SEQUENCE_{MIN,MAX,COUNT}` - Input sequence length histogram (defaults: 50.0, 128000.0, 12)
     /// - `DYN_METRICS_OUTPUT_SEQUENCE_{MIN,MAX,COUNT}` - Output sequence length histogram (defaults: 50.0, 32000.0, 10)
     /// - `DYN_METRICS_TTFT_{MIN,MAX,COUNT}` - Time to first token histogram (defaults: 0.001, 480.0, 18)
-    /// - `DYN_METRICS_ITL_{MIN,MAX,COUNT}` - Inter-token latency histogram (defaults: 0.001, 2.0, 13)
+    /// - `DYN_METRICS_ITL_{MIN,MAX,COUNT}` - Inter-token latency histogram (defaults: 0.001, 80.0, 20)
     /// - `DYN_METRICS_EMBEDDING_LATENCY_{MIN,MAX,COUNT}` - End-to-end `/v1/embeddings` latency histogram (defaults: 0.001, 10.0, 14)
     ///
     /// ## Model Configuration Metrics
@@ -1024,7 +1024,7 @@ impl Metrics {
 
         // Inter-token latency buckets: configurable via DYN_METRICS_ITL_{MIN,MAX,COUNT}
         let (itl_min, itl_max, itl_count) =
-            parse_bucket_config(env, env_metrics::DYN_METRICS_ITL, 0.001, 2.0, 13);
+            parse_bucket_config(env, env_metrics::DYN_METRICS_ITL, 0.001, 80.0, 20);
         let inter_token_latency_buckets = generate_log_buckets(itl_min, itl_max, itl_count);
 
         let inter_token_latency = HistogramVec::new(
@@ -2780,9 +2780,11 @@ mod tests {
 
     #[test]
     fn itl_ceiling_env_var_reaches_the_exported_le_labels() {
+        // Deliberately not the default (80.0, 20): a test that sets the env to the
+        // default value would pass even if the variable were ignored entirely.
         let env = fake_env(&[
-            ("DYN_METRICS_ITL_MAX", "80"),
-            ("DYN_METRICS_ITL_COUNT", "20"),
+            ("DYN_METRICS_ITL_MAX", "30"),
+            ("DYN_METRICS_ITL_COUNT", "8"),
         ]);
         let registry = Registry::new();
         let metrics = Metrics::build(None, &env);
@@ -2802,9 +2804,47 @@ mod tests {
         );
         assert_eq!(
             bounds.last().copied(),
-            Some(80.0),
+            Some(30.0),
             "top finite bucket should follow DYN_METRICS_ITL_MAX, got {bounds:?}"
         );
+        assert_eq!(
+            bounds.len(),
+            8,
+            "bucket count should follow DYN_METRICS_ITL_COUNT"
+        );
+    }
+
+    #[test]
+    fn itl_default_buckets_reach_80_seconds() {
+        // The ceiling was 2.0s, so any inter-token latency above it landed in `+Inf` and
+        // `histogram_quantile` pinned p99 at exactly 2.0 -- indistinguishable from a real
+        // 2s measurement. Guard the shipped default, not just the env-var override.
+        let registry = Registry::new();
+        let metrics = Metrics::build(None, &fake_env(&[]));
+        metrics.register(&registry).unwrap();
+        metrics
+            .inter_token_latency
+            .with_label_values(&["m"])
+            .observe(0.5);
+
+        let bounds = bucket_upper_bounds(
+            &registry,
+            &format!(
+                "{}_{}",
+                name_prefix::FRONTEND,
+                frontend_service::INTER_TOKEN_LATENCY_SECONDS
+            ),
+        );
+        // The exact set is the contract. Note the bottom edge of 0.0018: vLLM's equivalent
+        // starts at 0.01 and cannot resolve anything faster than 10ms per token, so raising
+        // the ceiling must not cost the low-end resolution Dynamo has and vLLM does not.
+        assert_eq!(
+            bounds,
+            vec![
+                0.0, 0.0018, 0.0033, 0.0059, 0.011, 0.02, 0.035, 0.064, 0.12, 0.21, 0.38, 0.69,
+                1.2, 2.3, 4.1, 7.4, 13.0, 24.0, 44.0, 80.0
+            ],
+        );
     }
 
     #[test]
@@ -2841,11 +2881,12 @@ mod tests {
     #[test]
     fn test_all_buckets_are_two_sig_figs() {
         let test_cases = vec![
-            (1.0, 256.0, 10),
+            (1.0, 512.0, 10),
             (50.0, 128000.0, 12),
             (50.0, 32000.0, 10),
             (0.001, 480.0, 18),
-            (0.001, 2.0, 13),
+            (0.001, 80.0, 20),
+            (0.001, 10.0, 14),
         ];
 
         for (min, max, count) in test_cases {