Support hierarchical namespace in Azure #3347

dimas-b · 2026-01-02T17:03:42Z

Add hierarchical to AzureStorageConfigInfo (the default is unset translating to current behaviour).
Use DataLakeDirectoryClient instead of DataLakeFileSystemClient for generating SAS tokens when hierarchical is set to true.
Add cloudTest classes for testing with credential vending in ADLS.

Checklist

🛡️ Don't disclose security issues! (contact [email protected])
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

evindj

Overall looks good to me, I do think we need to clarify the behavior when the feature is enabled but the account on azure does not have hierarchical namespace enabled.

evindj · 2026-01-02T17:57:06Z

spec/polaris-management-service.yml

+              description: >-
+                If set to `true`, instructs Polaris Servers to scope SAS tokens down to the most specific path
+                in the storage container (in most cases the table's base location). This flag should be set only
+                if hierarchical namespace is enabled in the Azure storage account.


What is the behavior if the flag is set but the Azure account does not have the feature enabled?
I am wondering if in a future iteration there will be a way to enable this feature based on whether or not the feature is enabled in Azure.

evindj · 2026-01-02T17:59:26Z

runtime/service/src/test/java/org/apache/polaris/service/entity/CatalogEntityTest.java

    CatalogEntity catalogEntity =
        new CatalogEntity.Builder()
-            .setName("testAwsConfigRoundTrip")
+            .setName("testStorageConfigRoundTrip")


+1 for removing AWS specific here.

evindj · 2026-01-02T19:06:21Z

.../src/main/java/org/apache/polaris/core/storage/azure/AzureCredentialsStorageIntegration.java

+            "Allowed read locations must not have more that one entry");
+        Preconditions.checkArgument(
+            allowedWriteLocations.size() <= 1,
+            "Allowed write locations must not have more that one entry");


N00b question for my own understanding, what is the use case for several allowedReadLocations and allowedWriteLocations?
Also why does this apply only to hierarchical use case?

what is the use case for several allowedReadLocations and allowedWriteLocations?

TBH, I do not really know 😅 Currently, only one location is passed in.

Azure can restrict SAS to only location (base directory or specific file).

dimas-b · 2026-01-05T15:32:39Z

@evindj :

do think we need to clarify the behavior when the feature is enabled but the account on azure does not have hierarchical namespace enabled.

Good point! Thanks for flagging it. I'll update the description in YAML.

snazy

The change looks good overall!

Please do not forget to update the CLI as well.

I was wondering whether we could implement this without introducing a configuration flag, but could not find a (cheap) way to figure out whether a file-system is hierarchical or not. So I guess there's no (performance neutral) way beside the introduced config flag.

.../src/main/java/org/apache/polaris/core/storage/azure/AzureCredentialsStorageIntegration.java

snazy · 2026-01-07T11:54:51Z

.../src/main/java/org/apache/polaris/core/storage/azure/AzureCredentialsStorageIntegration.java

+        return new DataLakePathClientBuilder()
+            .endpoint(endpoint)
+            .fileSystemName(fileSystemNameOrContainer)
+            .pathName(path) // TODO: drop authority part


jbonofre

LGTM, thanks !

snazy · 2026-01-09T07:55:08Z

...-core/src/main/java/org/apache/polaris/core/storage/azure/AzureStorageConfigurationInfo.java


+  /** The flag indicating whether the storage account supports hierarchical namespaces. */
+  @Nullable
+  public abstract Boolean isHierarchical();


Suggested change

public abstract Boolean isHierarchical();

@JsonInclude(JsonInclude.Include.NON_NULL)

public abstract Boolean isHierarchical();

Could actually also be (primitive):

Suggested change

public abstract Boolean isHierarchical();

@JsonInclude(JsonInclude.Include.NON_DEFAULT)

public abstract boolean isHierarchical();

Both help with serialization size and backwards compatiblity.

Not a blocker tho

An object Boolean is meant to allow distinguishing unset from false in the (generated) AzureStorageConfigInfo class. A simple boolean would be exposed to clients even it it was not explicitly set.

I'd rather not change AzureStorageConfigInfo serialization in this PR :)

Include.NON_DEFAULT is implied:

polaris/polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageConfigurationInfo.java

Line 95 in 992a428

JsonInclude.Include.NON_NULL, JsonInclude.Include.NON_NULL))

* Add `hierarchical` to `AzureStorageConfigInfo` (the default is unset translating to current behaviour). * Use `DataLakeDirectoryClient` instead of `DataLakeFileSystemClient` for generating SAS tokens when `hierarchical` is set to `true`. * Add `cloudTest` classes for testing with credential vending in ADLS.

dimas-b · 2026-01-09T18:51:46Z

rebased to resolve CHANGELOG.md conflicts

github-project-automation bot added this to Basic Kanban Board Jan 2, 2026

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jan 2, 2026

evindj reviewed Jan 2, 2026

View reviewed changes

dimas-b force-pushed the hierarchical-sas branch from 87e57ea to 35afeda Compare January 5, 2026 16:09

snazy reviewed Jan 7, 2026

View reviewed changes

jbonofre previously approved these changes Jan 7, 2026

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jan 7, 2026

dimas-b dismissed jbonofre’s stale review via 2ea6475 January 8, 2026 16:42

dimas-b force-pushed the hierarchical-sas branch from 35afeda to 2ea6475 Compare January 8, 2026 16:42

dimas-b requested a review from snazy January 8, 2026 16:43

snazy reviewed Jan 9, 2026

View reviewed changes

dimas-b added 4 commits January 9, 2026 13:18

review: clarify behaviour with non-hierarchical storage

56ef418

review: remove local debugging log

a984fce

review: remove irrelevant TODO comment

c24fce3

dimas-b force-pushed the hierarchical-sas branch from 2ea6475 to c24fce3 Compare January 9, 2026 18:51

	public abstract Boolean isHierarchical();
	@JsonInclude(JsonInclude.Include.NON_NULL)
	public abstract Boolean isHierarchical();

	public abstract Boolean isHierarchical();
	@JsonInclude(JsonInclude.Include.NON_DEFAULT)
	public abstract boolean isHierarchical();

Support hierarchical namespace in Azure #3347

Are you sure you want to change the base?

Support hierarchical namespace in Azure #3347

Conversation

dimas-b commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

evindj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Jan 5, 2026

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbonofre left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dimas-b commented Jan 2, 2026 •

edited

Loading