Extend S3 filesystem support in Gluten

[ReemaAlzaid]

Description

Add a Gluten S3 filesystem extension layer by extending Velox S3 classes

The goal is to enable enhanced S3 behavior directly in Gluten while still reusing Velox’s default S3 functionality

Proposed direction:

class GlutenS3FileSystem
    : public facebook::velox::filesystems::S3FileSystem

Gluten version

main branch

[ReemaAlzaid]

Velox S3 Internals Gluten May Need To Change

If Gluten needs deeper S3 customization, especially custom write behavior or async multipart upload, these Velox classes/internal APIs may need small upstream changes

Velox Class / Internal	Current Access Problem	Location	Why Gluten May Need It
S3FileSystem::Impl	Forward declared as protected, but methods are defined only in .cpp	S3FileSystem.h:91, S3FileSystem.cpp:230	Gluten can see impl_, but cannot call impl_->s3Client() or impl_->getS3Config() from a separate file
S3FileSystem::Impl::s3Client()	.cpp only internal method	S3FileSystem.cpp:439	Needed if Gluten needs to construct custom GlutenS3WriteFile
S3FileSystem::Impl::getS3Config()	.cpp only internal method	S3FileSystem.cpp:451	Needed if Gluten needs to pass S3 config into custom GlutenS3WriteFile
S3WriteFile::Impl	Forward declared as protected, implementation only in .cpp	S3WriteFile.h:80, S3WriteFile.cpp:38	Gluten cannot customize upload internals by subclassing
S3WriteFile::Impl::putObjectRequest()	.cpp only internal method	S3WriteFile.cpp:105	May be needed to customize single object upload behavior
S3WriteFile::Impl::upload(...)	Private internal method	S3WriteFile.cpp:221	May be needed to customize how data is split into multipart chunks
S3WriteFile::Impl::uploadPart(...)	Private internal method	S3WriteFile.cpp:239	May be needed for async/concurrent multipart upload
S3WriteFile::Impl::closed()	Private internal method	S3WriteFile.cpp:206	May be needed if Gluten owns upload state handling
S3ReadFile::Impl	Private PIMPL	S3ReadFile.h:67, S3ReadFile.cpp:41	Only relevant if Gluten later customizes S3 read behavior
S3ReadFile::Impl::preadInternal(...)	.cpp only internal method	S3ReadFile.cpp:143	Only relevant if Gluten later customizes actual S3 GET reads

cc: @FelixYBW

[FelixYBW]

Thank you, @ReemaAlzaid can you draft the PR?

cc @rui-mo

[ReemaAlzaid]

@FelixYBW Sure, here’s the initial draft PR: #12026. Should we build on top of it for the custom multi async upload path?

[FelixYBW]

Yes. It's the first PR for the customized S3 register. Can you draft velox PR as well to expose the necessary interface for customized Gluten one?

@weixiuli, the goal here is to implement Gluten's customized S3 file system, so we don't need to rely on Velox Community to review and merge Gluten-related enhancements.

Let's try to expose the necessary interface in Velox. If that doesn't work, we can copy the whole S3Filesystem code and pick a couple of PRs like 14472 into Gluten's S3 file system.

@zhli1142015 you may do similar things for abfs.

[ReemaAlzaid]

Yes. It's the first PR for the customized S3 register. Can you draft velox PR as well to expose the necessary interface for customized Gluten one?

Sure here's the velox PR for this change facebookincubator/velox#17400