[FEA] Add portable Spark application costing for YARN-based systems

[gerashegalov]

Is your feature request related to a problem? Please describe.

Spark benchmark results commonly report elapsed time, task duration, vcore-seconds, or executor-core-seconds. These metrics do not consistently represent the worker capacity allocated to an application or its cost.

YARN may schedule containers using memory with DefaultResourceCalculator, while Spark executor cores describe task concurrency. With DominantResourceCalculator, memory, vcores, GPUs, and other resources may affect the allocated node share.

We need a portable tool that attributes worker capacity and cost to Spark applications running on YARN-based platforms, initially covering:

• Amazon EMR
• Google Cloud Dataproc
• On-premises Hadoop/YARN clusters

The tool should enable consistent CPU/GPU benchmark comparisons across providers and cluster configurations.

Describe the solution you'd like

Add a command-line tool that calculates resource-proportional Spark application cost from Spark event logs and YARN logs.

The common accounting engine should:

• Use YARN allocation and terminal events as the authoritative source for container lifetimes and allocated resources.
• Use Spark event logs for application selection, metadata, task-duration diagnostics, and executor-to-container mapping.
• Detect the active YARN ResourceCalculator.
• Use memory share for DefaultResourceCalculator.
• Use the maximum allocated share across memory, vcores, GPUs, and supported custom resources for DominantResourceCalculator.
• Exclude ApplicationMaster containers by default.
• Preserve separate cost terms for heterogeneous worker types.
• Produce node-equivalent seconds before applying platform-specific prices.
• Report wall-clock time separately from attributed worker consumption.
• Mark incomplete accounting explicitly and avoid reporting final dollar values when completeness checks fail.
• Support CSV and JSON output for individual runs and baseline/test comparisons.
• Report task-duration and perfect-packing diagnostics without treating them as authoritative YARN cost.

Platform-specific adapters should handle log discovery, node classification, and optional pricing. Each adapter should:

• Locate or accept archived Spark and YARN logs.
• Identify worker-node types and advertised resource capacities.
• Resolve platform-specific resource names into stable node classes.
• Optionally provide hourly worker rates.
• Allow resource accounting without dollar conversion when pricing is unavailable.
• Preserve pricing inputs and lookup timestamps for auditability.

Initial adapters should cover Amazon EMR, Google Cloud Dataproc, and on-premises YARN deployments.

Proposed architecture:

Spark event logs
  └─ application selection, metadata, task diagnostics
                         |
YARN logs
  └─ allocations, resources, nodes, terminal events
                         |
                         v
             Common YARN accounting engine
                         |
          +--------------+---------------+
          |              |               |
          v              v               v
      Amazon EMR    Cloud Dataproc    On-prem YARN
        adapter          adapter          adapter
          |              |               |
          +--------------+---------------+
                         |
                         v
       CSV / JSON / comparison and audit output

Acceptance criteria:

• Common accounting logic is independent of any cloud provider.
• EMR, Dataproc, and on-premises inputs use documented adapters.
• Both default and dominant YARN resource calculators are supported.
• Memory, vcore, and GPU resource shares are covered by unit tests.
• Heterogeneous worker types remain separate in resource and cost expressions.
• Application selection and executor/container joins work with rolling Spark event logs.
• Completeness gates detect missing allocations, terminal events, nodes, or event-log segments.
• Pricing inputs and lookup timestamps are retained for auditability.
• Resource accounting works without access to pricing services.
• Tests use synthetic or sanitized fixtures.
• Documentation includes an example for each supported deployment type.

Describe alternatives you've considered

Spark event-log-only accounting: executor registration can occur after YARN allocation, executor removal events may be absent, and containers released before Spark registration may not appear. Spark events are valuable for metadata and diagnostics but are not a complete allocation ledger.

Raw YARN vcore-seconds: a YARN vcore is an accounting unit and may not be the resource that constrained scheduling. This can misrepresent node consumption under memory-based or dominant-resource scheduling.

Spark executor-core-seconds: executor cores describe task concurrency, not the node fraction allocated by YARN.

Separate tools for each platform: this would duplicate the core YARN parsing and accounting logic and could produce inconsistent definitions. Platform-specific behavior should be isolated behind adapters.

Additional context

The implementation should begin with a provider-neutral accounting model and interfaces for discovery and pricing adapters.

The initial contribution should include:

• Standardized resource and cost expressions.
• Synthetic test fixtures.
• Clear completeness and provenance reporting.
• No platform-specific assumptions in the common accounting engine.