GH-32123: [R] Expose azure blob filesystem by marberts · Pull Request #49553 · apache/arrow

marberts · 2026-03-19T03:08:36Z

Rationale for this change

This PR adds support for Azure. The Arrow R package already has support for AWS and GCS, and the Arrow C++ library has had support for Azure for a couple years now. Support for Azure is already available in pyarrow.

This would close #32123.

What changes are included in this PR?

A new class AzureFileSystem that's analogous to S3FileSystem/GcsFileSystem, along with a helper function az_container() that's analogous to s3_bucket()/gcs_bucket().
Updates to src/filesystem.cpp to interact with the machinery in arrow/filesystem/azurefs.h.
Updates to the configuration and build scripts to support building with support for Azure.
Updates to the vignettes on cloud storage, installation, and developer setup.

Are these changes tested?

Yes. See tests/testthat/test-azure.R.

Are there any user-facing changes?

Yes. There is a new function az_container(), serving the analogous role to s3_bucket()/gcs_bucket(), along with an R6 class AzureFileSystem, again serving the same role as S3FileSystem and GcsFileSystem. There is also a function arrow_with_azure() to indicate if Arrow was built with support for Azure.

GitHub Issue: [R] Expose Azure Blob Storage filesystem #32123

github-actions · 2026-03-19T03:08:59Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2026-03-19T03:09:44Z

⚠️ GitHub issue #32123 has been automatically assigned in GitHub to PR creator.

marberts · 2026-03-20T17:27:57Z

The CI is failing because the Azure C++ SDK depends on libxml2.

Update tools/nixlibs.R to check for libxml2 in with_cloud_support(), turning off the Azure feature flag if libxml2 isn't found.
- What about configure and configure.win?
- Is with_cloud_support() the only function that needs changing in tools/nixlibs.R?
Update the install.Rmd and developers/setup.Rmd vignettes to note that libxml2 is a dependency of Azure (same as with curl and openssl).
Update DESCRIPTION to include libxml2 in SystemRequirements.

marberts · 2026-03-24T02:16:12Z

It looks like the failing CI job is from a warning in R CMD check about a non-API call to a C function, due to all the changes in R-devel. I don't think this is due to our PR.

At this point, I'm fairly confident that we've implemented the Azure features correctly. The only thing I'm not sure about is the setup in configure, configure.win, and tools/nixlibs.R. The Azure C++ SDK used by Arrow requires libxml2, which is a new system dependency for this package. I don't fully understand how these configuration scripts work, so I wouldn't be surprised if they're not updated correctly to reflect the new system dependency.

@jonkeane, @thisisnic, @assignUser Please let me or @Collinbrown95 know if there are any changes needed to this PR.

thisisnic · 2026-03-24T15:43:49Z

Thanks for the PR @marberts - if you rebase from the main branch now, the non-API call stuff should be resolved.

Your best bet for the configure etc stuff is looking at the history of that file or older PRs to see how this kind of thing was handled there.

This PR where the GCS bindings were added might help: #13404, but let us know if you have any questions!

thisisnic · 2026-03-24T15:45:28Z

We'll also need additional CI jobs to test this - the PR I linked to above contains examples of what we added there.

marberts · 2026-03-24T19:52:50Z

Great, thanks @thisisnic!

marberts · 2026-03-24T19:53:37Z

To do list.

Rebase against main to fix failing CI job.
Review configure, configure.win and tools/nixlibs.R against changes in ARROW-16510: [R] Add bindings for GCS filesystem #13404. It's possible this will reveal some other changes that are needed.
Add CI job to test changes. See again ARROW-16510: [R] Add bindings for GCS filesystem #13404.

Collinbrown95 · 2026-03-26T03:55:46Z

@github-actions crossbow submit -g r

github-actions · 2026-03-26T03:56:18Z

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/23576717645

thisisnic · 2026-03-26T11:13:38Z

@github-actions crossbow submit -g r

github-actions · 2026-03-26T11:16:24Z

Revision: 4da7280

Submitted crossbow builds: ursacomputing/crossbow @ actions-92325d6543

Task	Status
r-binary-packages
r-recheck-most
test-r-alpine-linux-cran
test-r-arrow-backwards-compatibility
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-fedora-clang
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-offline-maximal
test-r-ubuntu-22.04
test-r-versions

marberts · 2026-03-29T01:07:11Z

@github-actions crossbow submit -g r

github-actions · 2026-03-29T01:07:47Z

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/23698292050

thisisnic · 2026-03-30T12:00:56Z

Some of those CI failures are from things which have been fixed on main - you'll need to rebase your branch.

…ript

The hypothesis is that the errors in the build logs are caused by incompatibilities between WIL (Windows Implementation Library) and MinGW GCC, so install them directly instead of trying to build from source.

…re C++ SDK

marberts · 2026-06-27T18:22:54Z

I figured out the rebase and upstream changes are now correctly reflected in this PR.

The big diff for github.packages.yml was due to a change in line ending from my editor; it's all good now and the diff makes sense.

jonkeane

Thank you for persisting with this. I know rebases can be a pain when the go sideways! A few questions / comments around installation + confirming that we have the right dependencies. The operative code looks ok, though I did not dig too too deeply on it. Do you happen to have a public azure bucket that I could point this at and try it out locally?

I'm also going to run crossbow to run the tests.

jonkeane · 2026-06-28T12:12:42Z

  # pkg-config --libs libcurl
  GCS_LIBS="-lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 \
            -lz -lws2_32 -lnghttp2 -ldbghelp"
+  # AZURE_LIBS="-lcurl -lssl -lxml2"


These are holdovers from when you were trying these on Windows, yeah?

Yes, I left these commented out in case progress gets made in the future around the incompatibility with mingw and the azure SDK stuff. Happy to either remove them or make it clearer in the comments.

jonkeane · 2026-06-28T12:19:13Z

      print_warning("requires libcurl-devel (rpm) or libcurl4-openssl-dev (deb)")
      arrow_s3 <- FALSE
      arrow_gcs <- FALSE
+      arrow_azure <- FALSE


Is libcurl and openssl necessary for Azure as well? These additions seem to indicate yes, but I want to confirm that

Although I did not test without openssl and libcurl, the azure c++ sdk requires them so I'm pretty sure they're needed.

jonkeane · 2026-06-28T12:20:19Z

+    } else if (!cmake_find_package("libxml2", NULL, env_var_list)) {
+      print_warning("requires libxml2-devel (rpm), or libxml2-dev (deb), libxml2 (brew)", "AZURE")
+      arrow_azure <- FALSE
    }


If I'm following correctly, this check will run regardless of if someone is trying to setup Azure. Is it possible to make this only run if someone has requested Azure?

Yes, good idea. Done in 1ea9150.

jonkeane · 2026-06-28T12:24:56Z

@github-actions crossbow submit -g r

github-actions · 2026-06-28T12:27:58Z

Revision: 004e725

Submitted crossbow builds: ursacomputing/crossbow @ actions-f9bf4c7b3a

Task	Status
r-binary-packages
r-recheck-most
test-r-alpine-linux-cran
test-r-arrow-backwards-compatibility
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-fedora-clang
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-offline-maximal
test-r-ubuntu-22.04
test-r-versions
test-r-wasm

marberts · 2026-06-28T19:56:08Z

Thank you for persisting with this. I know rebases can be a pain when the go sideways! A few questions / comments around installation + confirming that we have the right dependencies. The operative code looks ok, though I did not dig too too deeply on it. Do you happen to have a public azure bucket that I could point this at and try it out locally?

I'm also going to run crossbow to run the tests.

Awesome, thanks for taking a look.

Unfortunately I don't have a public container for Azure. If you want to try it locally you can use azurite: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite. This is what we're using in the test suite. The only other tests I did was to try connecting to a made up storage container to check that I was getting the expected errors.

…eing requested

marberts requested review from assignUser, jonkeane and thisisnic as code owners March 19, 2026 03:08

github-actions Bot added Component: R Component: Documentation awaiting review Awaiting review labels Mar 19, 2026

marberts changed the title ~~32123 expose azure blob filesystem~~ GH-32123 [R]: Expose azure blob filesystem Mar 19, 2026

marberts changed the title ~~GH-32123 [R]: Expose azure blob filesystem~~ GH-32123: [R] Expose azure blob filesystem Mar 19, 2026

github-actions Bot removed the Component: Documentation label Mar 19, 2026

marberts marked this pull request as draft March 25, 2026 23:32

github-actions Bot added Component: Parquet Component: C++ Component: Python Component: FlightRPC Component: Documentation labels Apr 1, 2026

Collinbrown95 and others added 15 commits June 26, 2026 21:42

wrap credential configuration methods with StopIfNotOk

f8e787d

move link flags to arrow_built_with ARROW_AZURE block in configure sc…

a1d72ab

…ript

add simple test function to work through codegen.R

82a5d37

cleanup azurefs test function code

246ff27

Trying PKGBUILD and windows build script again

1bdf2ab

Updated PKGBUILD

7148238

debug ci: add azure dependencies that CI tries to build from source.

d1c73bb

The hypothesis is that the errors in the build logs are caused by incompatibilities between WIL (Windows Implementation Library) and MinGW GCC, so install them directly instead of trying to build from source.

Removed tmp.md

8839eb2

Removed windows build and added macos

9639d0f

azure off for windows ci

cb1cb9f

Disable azure in configure.win

b367140

Add placeholder comment to flag present issues with MinGW and the Azu…

c4db209

…re C++ SDK

remove debugging script to install azure sdk from source

ce2e058

Updated vignettes to reflect Azure off on Windows

1279b02

Update github.packages.yml

c728602

marberts force-pushed the 32123-expose-azure-blob-filesystem branch from 46a4770 to c728602 Compare June 27, 2026 02:07

marberts added 4 commits June 26, 2026 22:23

Fixing lint in filesystem.cpp

4c370ca

Fixed warning in cpp code

df23828

Added build info for azure to tasks for mac + a couple code comments

739e455

Fixed line endings on github.packages yaml

004e725

jonkeane requested changes Jun 28, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jun 28, 2026

libxml2 warning in with_cloud_support() is now conditional on azure b…

1ea9150

…eing requested

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jun 29, 2026

Uh oh!

Conversation

marberts commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

marberts commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marberts commented Mar 24, 2026

Uh oh!

thisisnic commented Mar 24, 2026

Uh oh!

thisisnic commented Mar 24, 2026

Uh oh!

marberts commented Mar 24, 2026

Uh oh!

marberts commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Collinbrown95 commented Mar 26, 2026

Uh oh!

github-actions Bot commented Mar 26, 2026

Uh oh!

thisisnic commented Mar 26, 2026

Uh oh!

github-actions Bot commented Mar 26, 2026

Uh oh!

marberts commented Mar 29, 2026

Uh oh!

github-actions Bot commented Mar 29, 2026

Uh oh!

thisisnic commented Mar 30, 2026

Uh oh!

marberts commented Jun 27, 2026

Uh oh!

jonkeane left a comment

Choose a reason for hiding this comment

Uh oh!

jonkeane Jun 28, 2026

Choose a reason for hiding this comment

Uh oh!

marberts Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonkeane Jun 28, 2026

Choose a reason for hiding this comment

Uh oh!

marberts Jun 28, 2026

Choose a reason for hiding this comment

Uh oh!

jonkeane Jun 28, 2026

Choose a reason for hiding this comment

Uh oh!

marberts Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

jonkeane commented Jun 28, 2026

Uh oh!

github-actions Bot commented Jun 28, 2026

Uh oh!

marberts commented Jun 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

marberts commented Mar 19, 2026 •

edited

Loading

marberts commented Mar 20, 2026 •

edited

Loading

marberts commented Mar 24, 2026 •

edited

Loading

marberts Jun 28, 2026 •

edited

Loading