feat(logs): euler-parity golden-line sources + mapping config (on #2075) by kanikac199 · Pull Request #2076 · juspay/hyperswitch-prism

kanikac199 · 2026-08-07T10:02:08Z

Stacked on top of #2075 (config-driven log-field engine). Adds the euler source fields the app must emit so the config engine can rename/nest them, plus the actual euler field-mapping config.

What this adds

Code

http/error.rs: expose http_status_for_status() — the golden line reports the connector's exact 4xx/5xx status, not the coarse gRPC code.
Incoming line (utils.rs):
- res_code — numeric euler status (200 on success, connector-aware HTTP status on error), recorded additively via the Storage API. status_code is left untouched (see compatibility note below).
- latency_ms — additive numeric latency (euler latency is a number), recorded via a shared record_latency_ms() helper used by both the streaming and non-streaming wrappers.
- Euler id sources — merchant_order_id / customer_id (customer.id) / merchant_transaction_id extracted from the masked payload and recorded as flat span fields (the engine sources by flat key; it can't reach into request_body). Written via the storage API, so no per-handler #[instrument] edits.
- action — the real HTTP verb on the request span (gRPC is always POST; the HTTP gateway carries the true verb) → feeds api_details.method.
- request/response bodies + headers recorded as structured JSON (merged from feat(logs): Euler-compatible log structure with structured JSON fields #2094): record_json_fields_on_span + maskable_headers_to_json.

Config (development.toml)

[log.fields] with a runtime enabled toggle (default opt-out — only development.toml sets enabled = true; production.toml/sandbox.toml/docker_compose.toml have no [log.fields] block).
Per-direction [log.fields.incoming] / [log.fields.outgoing] tables. Each entry is one of:
- { source = "<span field>" } — copy/rename a flat span field. Dotted target keys (e.g. "api_details.api_tag") build the nested api_details object.
- { value = "<literal>" } — inject a constant (e.g. "api_details.req_type" = { value = "INTERNAL" } incoming / "EXTERNAL" outgoing).
Renders both golden lines in euler shape: flat latency / entity / udf_* keys plus the nested api_details object, split per direction.

Note: an earlier draft of this description referenced [log.transformations.incoming|outgoing] + [log.static_values]; the implemented schema is [log.fields.incoming|outgoing] with { source = … } / { value = … } (updated here so operators aren't misled about the TOML keys).

`status_code` compatibility — additive, not breaking

status_code is left exactly as main emits it: the gRPC code-name string (e.g. "InvalidArgument") on error, and unset (Empty) on success. The numeric euler value lives in a new additive res_code field (recorded via the Storage API, no #[instrument] edits), and the config sources resp_code / api_details.res_code from res_code. No existing consumer of status_code breaks.

Verified live: success → res_code=200, status_code absent; error → status_code="…" unchanged, res_code=<http status>.

Verified live

authorizedotnet (Authorize): incoming action="POST" → api_details.method="POST", category="INCOMING_API", udf_* populated, entity="Authorize", full nested api_details; outgoing category="OUTGOING_API", api_details.url = connector URL.
twoc_twop_paco (2C2P PACO, euler→UCS end-to-end): authorize (3DS) → charged → refund (pre-settlement VOID). Golden line carries api_details.{api_tag, req_type, method, latency, url}, latency_ms, entity, and structured req/res bodies. Real 2c2p sandbox responses (invoice CEBU…, PC-B050000 Success).

Notes

The api_details → message rename stays downstream (Vector/LP): euler reads message.*, but message is reserved in log_utils, so the app stages the object under api_details.
hostname/version/service need no mapping — euler derives _version/_service from the pod-name hostname (which log_utils auto-emits); euler has no standalone version field.
Build prereqs: rustc ≥ 1.96 and CARGO_NET_GIT_FETCH_WITH_CLI=true (the pinned log_utils rev is a bare sha).

…olden log lines Add Euler-compatible log schema support: - Config-driven field renaming for golden log lines via [log.transformations.incoming] and [log.transformations.outgoing] (gated behind log-transformations feature flag) - Runtime static key-value pairs via [log.static_values], overridable per-request through x-config-override header - transformation_mode (copy/replace) for event field transformations - Deadlock fix: filter reserved keys before entering span write lock to prevent re-entrant deadlock from log_utils record_value calling tracing::warn! Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

…ndexing_slicing

Allows different static key-value pairs for incoming (gRPC handler) and outgoing (connector call) golden log lines. Config format changes from flat [log.static_values] to [log.static_values.incoming] and [log.static_values.outgoing].

… and feature gating Replace separate log.transformations and log.static_values with a single log.fields config supporting Source (span lookup) and Value (literal) entries. Dotted target paths build nested JSON with deep-merge semantics. - Add LogFieldEntry enum, CompiledLogFields compilation, and apply_log_fields() - Add post_patch_processing() on Config for recompiling derived fields - Centralize compiled_outgoing: remove 18 duplicate compile calls, use pre-compiled config.log_fields directly from Config - Recompile log_fields in merge_config_with_override() after patch apply - Gate runtime application behind log-transformations feature flag - x-config-override header merges log.fields with extend (not replace) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ng HTTP verb - http/error.rs: expose http_status_for_status() so the golden line reports the connector's exact 4xx/5xx (not the coarse gRPC code) - incoming line records status_code (200 / connector-aware), the euler id sources (merchant_order_id / customer_id / merchant_transaction_id) as flat span fields, and action (real HTTP verb) so the log transforms can source them by flat key

Populate [log.fields.incoming|outgoing] (Amit's unified LogFieldEntry map) so both golden lines render in euler shape: flat udf_* / category / entity / resp_code keys plus the nested api_details object. { source = } reads a span field; { value = } is a literal. Per-request patchable via x-config-override.

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

euler sends the tenant name as x-tenant-id (UCS span tenant_id); map it to tenant_name so the name is preserved, while euler forwards the real tenant_id (uuid) per-request via x-config-override (overrides on merge).

…ffi build EventProcessingParams is ungated and always holds a &CompiledLogFields field, but the import was gated on injector-client. The ffi crate builds external-services with default-features=false (no injector-client), so the type was referenced but not imported (E0412). Move the import out of the injector-client cfg block so every feature combo compiles.

…ure gate

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

Update log_utils rev to use Cow<'static, str> keys in Storage instead of String. Adapt apply_log_fields snapshot type and record_value calls.

Add grpc-server/log-transformations to both nextest run commands so CI tests compile and run with log field transformation code enabled. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add `enabled` field to `LogFields` config (default: false) so log field application can be toggled at runtime via config or x-config-override without rebuilding. Also add log-transformations feature to Dockerfile. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Resolve conflicts by adopting main's runtime log-fields 'enabled' toggle (LogFields.enabled + LogFieldsPatch, log_fields_enabled threading through EventProcessingParams/log_after_initialization, gated apply_log_fields) and main's Cow<'static, str> storage-key refactor; keep the branch's golden-line sources/mapping config and the unconditional CompiledLogFields import. Dependency conflicts: align framework-libs-rs (log_utils/build_info) to main's rev 76613f5d in all Cargo.toml + Cargo.lock.

…ble log fields - add numeric `latency_ms` span field (incoming wrappers + outgoing connector step) so `latency`/`api_details.latency` are numbers like euler (existing string `response_time`/`latency` left untouched — additive) - emit `timestamp` alongside `time` in the JSON formatter (euler-schema alias) - `api_details.req_type` static value: INTERNAL (incoming) / EXTERNAL (outgoing) - set `[log.fields] enabled = true` (main made application opt-in)

The edited ucs_env/logger/formatter.rs is dead code (not wired into mod.rs); the active formatter is log_utils's, which emits `time`. The timestamp edit had no effect. The euler-schema `time`->`timestamp` rename is handled by the LP overlay instead. Keeps the working latency_ms/req_type/enabled changes.

…y unused_qualifications)

Use record_json_fields_on_span() to write request_body, response_body, request.headers, request.body, response.body, and response.headers as serde_json::Value directly into span storage. This produces proper nested JSON in log lines instead of escaped JSON strings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…og change, use masked_serialize for response_body

…t feature

…nse body + headers Combine with the branch's golden-line work: keep masked_body for the udf_* id extraction and numeric latency_ms, adopt #2094's record_json_fields_on_span / maskable_headers_to_json so request/response body + headers emit as nested JSON (not escaped debug strings).

Resolve conflicts in utils.rs (keep status_code=200 on success + masked_body binding for udf-id extraction) and service.rs (keep CompiledLogFields comment). Dedup the identical latency_ms recording into a shared record_latency_ms() helper (review #3).

…es_code Keep the `status_code` span field exactly as main (gRPC code-name string on error, Empty on success) and emit the numeric euler value under a new `res_code` field via the Storage API. Config now sources `resp_code`/`api_details.res_code` from `res_code`. No consumer of the existing `status_code` field breaks.

AmitsinghTanwar007 · 2026-08-13T19:09:16Z

+# Patchable per-request via `x-config-override` (merge semantics).
+[log.fields]
+# Runtime toggle (main made log-field application opt-in; default false).
+enabled = true
+
+[log.fields.incoming]
+"x-request-id" = { source = "request_id" }
+"udf_order_id" = { source = "merchant_order_id" }
+"udf_customer_id" = { source = "customer_id" }
+"udf_txn_uuid" = { source = "merchant_transaction_id" }
+"entity" = { source = "flow" }
+"resp_code" = { source = "res_code" }
+"category" = { value = "INCOMING_API" }
+"schema_version" = { value = "V2" }
+"tag" = { value = "euler_logs" }
+# euler sends the tenant NAME as x-tenant-id (span `tenant_id`); relabel it to `tenant_name`.
+# euler overrides the real `tenant_id` (uuid) per-request via x-config-override.
+"tenant_name" = { source = "tenant_id" }
+# euler emits `latency` at the top level (in addition to the nested one); mirror it here.
+"latency" = { source = "latency_ms" }
+"api_details.url" = { source = "uri" }
+"api_details.method" = { source = "action" }
+"api_details.res_code" = { source = "res_code" }
+"api_details.req_body" = { source = "request_body" }
+"api_details.res_body" = { source = "response_body" }
+"api_details.error" = { source = "error_message" }


this cant be in open source file you have to have this in your own deployment toml file

AmitsinghTanwar007 · 2026-08-13T19:10:05Z

+    // `CompiledLogFields` is used by the ungated `EventProcessingParams` struct, so it must be
+    // imported unconditionally (the ffi build compiles this crate without `injector-client`).


no need of this comment

AmitsinghTanwar007 · 2026-08-13T19:10:45Z

        latency = Empty,
+        latency_ms = Empty,


what is the difference between both the above fields

Both point to the same elapsed time. latency is the human-readable string we already had; latency_ms is the exact same duration but as a plain number of milliseconds, so downstream consumers that want a numeric latency can read it directly instead of parsing the string. One is the string form, the other the numeric form.

AmitsinghTanwar007 · 2026-08-13T19:17:11Z


    let elapsed = start.elapsed().as_millis();
    tracing::Span::current().record("latency", elapsed);
+    // Additive numeric latency (euler `latency` is a number). `latency` (string) left as-is.


dont mention euler in comments

AmitsinghTanwar007 · 2026-08-13T19:19:24Z

    let elapsed = start.elapsed().as_millis();
    tracing::Span::current().record("latency", elapsed);
+    // Additive numeric latency (euler `latency` is a number). `latency` (string) left as-is.
+    tracing::Span::current().record("latency_ms", u64::try_from(elapsed).unwrap_or(u64::MAX));


is there any need of this because we have time out and i dont think so it will ever cross 18,446,744,073,709,551,615

Agreed it will never realistically overflow. Duration::as_millis() returns a u128 and the workspace lints forbid as casts and .unwrap(), so I still need a checked conversion to u64. I dropped the misleading u64::MAX fallback and switched to unwrap_or_default() (i.e. 0), so on the impossible overflow it is just 0 rather than a fake max value.

AmitsinghTanwar007 · 2026-08-13T19:20:20Z

+        // euler's `action` = the real HTTP verb (GET/POST/…); euler maps it to the `_method`
+        // column. gRPC-over-HTTP2 is always POST, the HTTP gateway carries the true verb.


dont mention euler avoid this comments

AmitsinghTanwar007 · 2026-08-13T19:24:33Z

+    if let Some(ids) = masked_body.as_ref().map(|body| {
+        [
+            (
+                "merchant_order_id",
+                body.get("merchant_order_id").and_then(Value::as_str),
+            ),
+            (
+                "customer_id",
+                body.get("customer")
+                    .and_then(|customer| customer.get("id"))
+                    .and_then(Value::as_str),
+            ),
+            (
+                "merchant_transaction_id",
+                body.get("merchant_transaction_id").and_then(Value::as_str),
+            ),
+        ]


the two merchant id are euler specific id's if they know the values they can send it in runtime headers?

These are not extra euler-only values. merchant_order_id, customer.id and merchant_transaction_id are already part of the standard request body UCS receives for every payment. I am just reading them out of the body we already have and putting them on the span as flat fields so the log mapping can pick them up (it cannot reach into the nested request_body). So the caller does not need to resend them as headers, we already have them. I reworded the comment to make that clear.

its ok they are part of request body
but if you need them in logging can't you use the header of log transformation?

AmitsinghTanwar007 · 2026-08-13T19:25:42Z

+        log_utils::Storage::with_current_span_mut(|storage| {
+            for (key, value) in ids {
+                if let Some(value) = value {
+                    storage.record_value(key, Value::String(value.to_owned()));
+                }
+            }
+        });
+    }


dont call this function instead use the existing function record_json_fields_on_span

AmitsinghTanwar007 · 2026-08-13T19:27:09Z

+            log_utils::Storage::with_current_span_mut(|storage| {
+                storage.record_value("res_code", Value::from(200_i64));
+            });


use the existing function

AmitsinghTanwar007 · 2026-08-13T19:35:45Z

+            let http_status = crate::http::error::http_status_for_status(status).as_u16();
+            log_utils::Storage::with_current_span_mut(|storage| {
+                storage.record_value("res_code", Value::from(i64::from(http_status)));
+            });


use the existing function

AmitsinghTanwar007 · 2026-08-13T19:36:32Z

+fn record_latency_ms(duration: u128) {
+    log_utils::Storage::with_current_span_mut(|storage| {
+        storage.record_value(
+            "latency_ms",
+            Value::from(u64::try_from(duration).unwrap_or(u64::MAX)),
+        );
+    });
+}


is this needed

It is a small helper but it is called from two places (the streaming and non-streaming logging wrappers), so it exists to keep those two identical and avoid drift. I have simplified it to just call record_json_fields_on_span, so it is now a 3-line wrapper. Happy to inline it at both call sites if you prefer.

…toml, reuse record_json_fields_on_span, scrub comments - Remove the euler-specific [log.fields] mapping from the open-source config/development.toml (it now lives in the deployment toml). The mechanism stays; the mapping is deployment config. - Use the existing record_json_fields_on_span helper instead of poking log_utils::Storage directly for res_code, latency_ms, and the request identifiers. - Drop the saturating u64::MAX latency fallback (unwrap_or_default). - Remove connector-name-specific mentions from comments; clarify that the order/customer/transaction ids are standard request fields and that latency vs latency_ms is string-vs-numeric.

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

AmitsinghTanwar007 and others added 4 commits August 7, 2026 10:30

chore: auto-fix formatting and generated code

9a45786

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

fix(events): replace indexing with safe .get()/.first() for clippy::i…

0d7f308

…ndexing_slicing

kanikac199 requested review from a team as code owners August 7, 2026 10:02

kanikac199 force-pushed the feat/euler-log-schema-on-structure branch from 894aa60 to 0696dff Compare August 7, 2026 10:41

AmitsinghTanwar007 and others added 4 commits August 9, 2026 20:54

fix(logs): restore CompactJson log format (revert testing change)

5bab766

kanikac199 force-pushed the feat/euler-log-schema-on-structure branch from 0696dff to 600da90 Compare August 10, 2026 07:21

hyperswitch-bot Bot and others added 10 commits August 10, 2026 07:23

chore: auto-fix formatting and generated code

c8f44ff

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

feat(config): relabel tenant_name from x-tenant-id name

4745df1

euler sends the tenant name as x-tenant-id (UCS span tenant_id); map it to tenant_name so the name is preserved, while euler forwards the real tenant_id (uuid) per-request via x-config-override (overrides on merge).

Merge branch 'main' into feat/euler-log-structure

9a7da74

Merge branch 'main' into feat/euler-log-structure

7dc5933

fix(logs): move CompiledLogFields import outside injector-client feat…

76bc7b1

…ure gate

chore: auto-fix formatting and generated code

b2e6525

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

refactor(deps): update framework-libs-rs to Cow-based Storage keys

0a19920

Update log_utils rev to use Cow<'static, str> keys in Storage instead of String. Adapt apply_log_fields snapshot type and record_value calls.

ci: enable log-transformations feature in test commands

248c3c5

Add grpc-server/log-transformations to both nextest run commands so CI tests compile and run with log field transformation code enabled. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Base automatically changed from feat/euler-log-structure to main August 11, 2026 12:33

kanikachaudhary199 and others added 7 commits August 11, 2026 19:42

fix(logs): drop redundant serde_json:: qualifier on latency_ms (clipp…

7df4b57

…y unused_qualifications)

Merge branch 'main' into feat/euler-log-structure

7b83411

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

refactor(logs): extract maskable_headers_to_json util, revert event l…

80f9c10

…og change, use masked_serialize for response_body

AmitsinghTanwar007 and others added 6 commits August 12, 2026 01:23

fix(logs): gate maskable_headers_to_json import behind injector-clien…

f59f02c

…t feature

fix(logs): move record_json_fields_on_span import to unconditional block

d49d062

fix(logs): use standard masking format for masked header values

b510b43

Aishwariyaa-Anand assigned kanikac199 Aug 13, 2026

AmitsinghTanwar007 reviewed Aug 13, 2026

View reviewed changes

AmitsinghTanwar007 requested changes Aug 13, 2026

View reviewed changes

kanikachaudhary199 and others added 2 commits August 14, 2026 14:13

chore: auto-fix formatting and generated code

6bb3894

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

Aishwariyaa-Anand approved these changes Aug 14, 2026

View reviewed changes

		// `CompiledLogFields` is used by the ungated `EventProcessingParams` struct, so it must be
		// imported unconditionally (the ffi build compiles this crate without `injector-client`).

		// euler's `action` = the real HTTP verb (GET/POST/…); euler maps it to the `_method`
		// column. gRPC-over-HTTP2 is always POST, the HTTP gateway carries the true verb.

Uh oh!

Conversation

kanikac199 commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

status_code compatibility — additive, not breaking

Verified live

Notes

Uh oh!

AmitsinghTanwar007 Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kanikac199 commented Aug 7, 2026 •

edited

Loading

`status_code` compatibility — additive, not breaking

AmitsinghTanwar007 Aug 13, 2026 •

edited

Loading