Supply a hint arrow schema for casting Parquet field types during scans#814
Supply a hint arrow schema for casting Parquet field types during scans#814gruuya wants to merge 1 commit intoapache:mainfrom
Conversation
This is so as to avoid a potential schema mismatch resulting from upcasting arrow 8 and 16 bit integers to Iceberg 32 bit integer type.
| if task.schema.as_struct().fields().iter().any(|field| { | ||
| matches!( | ||
| field.field_type.as_ref(), | ||
| Type::Primitive(PrimitiveType::Int) | ||
| ) | ||
| }) { |
There was a problem hiding this comment.
Maybe this should be done only if the field with this type is also one of the projected ones.
|
Making this a draft as the upstream dependency is also a draft atm. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
1 participant
This is so as to avoid a potential schema mismatch resulting from upcasting arrow 8 and 16 bit integers to Iceberg 32 bit integer type.
This is one way to resolve #813. Note that this is dependent on apache/arrow-rs#6892 getting merged (and picked up) first.
I still need to think of a proper test case for this too.
Closes #813.