[C++][Azure] Add ApplicationId to AzureFileSystem

[nateprewitt]

Describe the enhancement requested

This issue is proposing modification of the User Agent identifier used in calls from AzureFileSystem to Azure Storage backends. This addition is part of existing Azure C++ SDK functionality and will make it easier to identify requests coming from Arrow/PyArrow rather than a stand-alone C++ SDK.

Would the Arrow team be open to accepting a PR for this? I have example code here which is tested and should be ready to submit if this would be an acceptable enhancement.

The proposed code prepends a azpartner-arrow/{version} identifier to the HTTP User-Agent Header using the current build version. Here are some examples on precedent for updating other libraries/frameworks to use this format:

NIXL Azure Blob Client
Langchain Azure Storage

Component(s)

C++

[kou]

Could you open a PR from the branch?

I'm not sure azparter-arrow/ is suitable for our identifier. Where does azpartner- come from?

[nateprewitt]

I'm not sure azpartner-arrow/ is suitable for our identifier. Where does azpartner- come from?

I agree it's an odd designation. This covers any third-party tooling (not Microsoft authored) calling Azure. I've confirmed this is the intended identifier for a library like Arrow. Whether you want to add this though is up to you. It's primarily to help disambiguate tooling issues by adding a finer grained identifier.

Let me know what your thoughts are and I can put up the PR if you'd still like me to move forward.

[kou]

Is the azpartner- prefix widely used?

[raulcd]

Is azpartner- a recommendation from the Azure team for third party User Agent identifier?

[nateprewitt]

@kou @raulcd This is the established convention for 3p identifiers for Azure Storage, but I don't believe it's documented anywhere.

It's done in a handful of projects today and we're working on expanding coverage. Full disclosure, I'm submitting this proposal on behalf of Azure Storage so we have a clearer picture of usage (and where we focus OSS contributions).

[kou]

OK. Could you document the convention in the official Azure Storage documentation?

Can we use azpartner-apache-arrow not azpartner-arrow (don't omit "Apache")?

[nateprewitt]

I think the intent is to add this to high-usage libraries that have a direct Azure integration rather than to be a general purpose pattern. Inclusion is intended to be on an as-needed basis for tooling that drives high traffic on Azure Storage. That's the primary reason it's not publicly documented.

The current limitation on the ApplicationId is 24 characters. If there are branding concerns about telemetry naming from Apache, I can raise those. The currently proposed name is to work within that constraint.

[nateprewitt]

To add some more data, this is what I'm seeing elsewhere in the Apache ecosystem. Having a combination of integration + product name without the apache prefix has some precedent. I'm not trying to say this is the right decision for Arrow, but providing some examples for how the larger Apache ecosystem is currently handling these use cases.

Apache Iceberg Snowflake Integration - iceberg-snowflake-catalog/
Apache Airflow Databricks Integration - databricks-airflow/
Apache Airflow S3 Integration - Airflow/
Apache Beam Google Cloud Integration - (GPN:Beam) (I believe this is Google Partner Network, but also similarly undocumented)

[raulcd]

Given the 24 characters limitation azpartner-arrow/{version} sounds reasonable to me, @kou do you agree?

[kou]

OK. Let's use azpartner-arrow/{version}.

In general, "Apache Arrow" is better than "Apache": https://www.apache.org/foundation/marks/guide

But we have the 24 characters limitation in this case.

[nateprewitt]

Great to hear! I've opened #49301 to add the ApplicationId. There is currently no test on the branch, but I've manually validated everything works as expected.

If we do want a test, what granularity would you be looking for? That we're assembling the string correctly or would we want something built against Azurite to validate the SDK behavior? If you don't feel that's necessary, I'll leave the branch as-is.