Skip to content

Conversation

cetra3
Copy link

@cetra3 cetra3 commented Oct 3, 2025

This PR adjusts the way that dictionary values are handled. Before this PR, they were broken in the following ways:

  • The FlightDataEncoder by default will Hydrate dictionary values, hoisting their values rather than encoding dictionaries. This wasn't handled correctly in the network_ execution plans, that were expecting the schema to not change, and so adding some schema adapters resolved this
  • The metrics data did not handle dictionaries very well either when constructing an empty record batch, as it needs the dictionary schema to be loaded. However, as we've already written the stream using FlightDataEncoder, this means that the dictionary ids can be all messed up.

Rather than pfaff about with all that side of things, I've adjusted it to be a "peek" style stream, whereby we wait until the last flight data, and add the metrics to that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant