fix(substrait): Do not add implicit groupBy expressions when building logical plans from Substrait

[anlinc]

Which issue does this PR close?

Closes #14348

Rationale for this change

Substrait plans are intended to be interpreted literally. When you see plan nodes like:

"project": {
  "common": {
    "emit": {
      "outputMapping": [0, 3]
    }
  },
...
}

The output mapping (e.g. [0, 3]) contains ordinals representing the offset of the target expression(s) within the [input, output] list. If the DataFusion LogicalPlanBuilder is introducing additional input expressions, this violates the plan's intent and will produce the incorrect output mappings. Please see the issue for a concrete example.

What changes are included in this PR?

In the Substrait path, do not add additional grouping expressions derived from functional dependencies.

Are these changes tested?

Added a multilayer aggregation Substrait example. The first aggregation produces a unique column with a functional dependency. Despite this, the second aggregation must not introduce any additional grouping expressions.

There should be no changes in the non-Substrait path.

Are there any user-facing changes?

No.

[anlinc]

Super new to Rust -- is this an okay / conventional way to name private helpers?

[Blizzara]

I don't think there's need for _ since the function is already private (by virtue of not being pub fn). Something like aggregate_inner I think is used quite a lot.

Alternatively, given the logicalplanbuilder for aggregate doesn't do that much, we could also just inline it into the substrait consumer. That way it's not changing the LogicalPlanBuilder api, which might be easier.

Or maybe this whole add_group_by_exprs_from_dependencies thing should move from the plan builder into the analyzer/optimizer? Intuitively it feels like the constructed logical plan shouldn't do this kind of magic, but the analyzer/optimizer can if it makes things faster to execute. But that might be a bigger undertaking, so I'd be quite fine with this PR or the alternative above first.

[Blizzara]

was this succeeding also before? (I'd guess so as it'd add the extra groupbys but take that into consideration while producing the plan, is that right?)

[Blizzara]

nit: style-wise I'd prefer keeping group_expr non-mut and doing something like:

Suggested change

	if include_implicit_group_by_exprs {
	group_expr =
	add_group_by_exprs_from_dependencies(group_expr, self.plan.schema())?;
	}
	let group_expr = if include_implicit_group_by_exprs {
	group_expr =
	add_group_by_exprs_from_dependencies(group_expr, self.plan.schema())?;
	} else {
	group_exrp
	};

[Blizzara]

Thanks, seems like a clear enough bug, appreciate both the report and the PR to fix it!