Improved N-1 join query performance for DW SQL #2631

lingxiao-microsoft · 2025-03-21T23:41:37Z

Why make this change?

Back 1 year ago, as DW SQL does not support JSON PATH for converting the execution to json format. Hence we had to use STRING_AGG as the workaround.
Recently, we've noticed JSON PATH is now supported for outer query in DW, and we can use JSON_OBJECT + JSON_PATH to address the json conversion for N-to-1 relations, which can optimize the performance.
For N-N relations, we're still looking into any resolutions as JSON_ARRAYAGG does not provide much performance improvements.
For other scenarios when joins are not needed for a simple SELECT, we will JSON PATH instead of STRING_AGG for better performance.

What is this change?

This PR covers

Introduced a feature flag to safeguard the changes, the feature flag is default as False when not provided to avoid any regressions. It will be removed once the changes are validated in production with scoped audiences.
For DW query builder, use JSON_OBJECT to generate the columns for sub-queries and applied JSON PATH to handle outer query, which fully replace the need of STRING_AGG.
Also, for non-join queries (in which we don't need to handle the relations), used JSON PATH to replace the need of STRING_AGG for better performance as well. This will have impact on aggregations, non-join queries and pagination.
Added some helper functions into the unit tests module, which aims to compare the results from GraphQL & DB engine easily for deeply nested queries.

How was this tested?

Unit Tests
- As this change does not introduce new scenarios, so mostly added some new test cases to get more coverage when M-M / M-1 join queries are needed.
Integration Tests

Manual Testing - Join Scenarios

Query 1-1 relation - As expected, optimization applied

Query N-1 relation - As expected, optimization applied

Query 1-N Relation - As expected, optimization not applied

Query N-N Relation - As expected, optimization not applied

Other Scenarios

We've applied the JSON PATH when there is no join in the query to replace the STRING_AGG for better performance.

Aggregation

Non-Join Query

Pagination

N to 1, total items: 3
N to N

src/Core/Resolvers/DWSqlQueryBuilder.cs

src/Service.Tests/SqlTests/GraphQLQueryTests/DwSqlGraphQLQueryTests.cs

vadeveka · 2025-03-26T19:50:59Z

src/Service.Tests/SqlTests/GraphQLQueryTests/DwSqlGraphQLQueryTests.cs

+        /// </summary>
+        /// <returns></returns>
+        [TestMethod]
+        public async override Task DeeplyNestedManyToOneJoinQuery()


Should we be having JSON PATH applied in this case for the outer query?

Yes, but the idea here is to do one step further to make sure the results we generated using JSON PATH is same as the results generated previously using STRING AGG to ensure backward compatibility.

rohkhann · 2025-03-26T21:44:24Z

src/Cli.Tests/ModuleInitializer.cs

@@ -53,6 +53,10 @@ public static void Init()
        VerifierSettings.IgnoreMember<RuntimeConfig>(options => options.EnableAggregation);
        // Ignore the EnableAggregation as that's unimportant from a test standpoint.
        VerifierSettings.IgnoreMember<GraphQLRuntimeOptions>(options => options.EnableAggregation);
+        // Ignore the EnableDwNto1JoinOpt as that's unimportant from a test standpoint.
+        VerifierSettings.IgnoreMember<RuntimeConfig>(options => options.EnableDwNto1JoinOpt);


Adding to runtime config makes this publicly facing and something users should be aware of @JerryNixon on his inputs here.

rohkhann · 2025-03-26T21:45:31Z

src/Config/ObjectModel/GraphQLRuntimeOptions.cs

@@ -11,7 +11,8 @@ public record GraphQLRuntimeOptions(bool Enabled = true,
                                    bool AllowIntrospection = true,
                                    int? DepthLimit = null,
                                    MultipleMutationOptions? MultipleMutationOptions = null,
-                                    bool EnableAggregation = true)
+                                    bool EnableAggregation = true,
+                                    bool EnableDwNto1JoinOpt = false)


Dab users would not see the benefits of this perf improvement and it is a very unintuitive thing to go to config file and enable. @aaronburtle any other way of having a feature flag apart from adding it in the runtime config?

We can flip this flag to always true once we've verified with customers. So that everyone else can benefit from it without flip it in config file

@aaronburtle , pls let me know if there is a better way to achieve the purpose without making it public configuration to everyone

DAB users will benefit if they use dwsql with Synapse or fabric warehouses/SAE. It doesnt need to be a public configuration though. We should enable this optimization by a default after building confidence in GraphQL workload.

{ "feature-flags": { "enable-sqldw-join-rewrite": true } "runtime": { ... } }

Great. Let's not document these or add them to dab validate or dab config. I think feature flags are temporary and might be removed later. So, docs would confuse the matter. I think updating JSON schema for feature-flags still makes sense so schema validation can still pass. Meaning, allowing /anything/ inside feature-flags.

latest update: looks like we dont necessarily need this option altogether in the config/schema. So, @lingxiao-microsoft is looking at if he can get away without adding the feature-flags section altogether

rohkhann · 2025-03-26T21:46:23Z

src/Core/Resolvers/DWSqlQueryBuilder.cs

@@ -15,8 +15,17 @@ namespace Azure.DataApiBuilder.Core.Resolvers
    public class DwSqlQueryBuilder : BaseSqlQueryBuilder, IQueryBuilder
    {
        private static DbCommandBuilder _builder = new SqlCommandBuilder();
+        private readonly bool _enableNto1JoinOpt;
+        private const string FOR_JSON_SUFFIX = " FOR JSON PATH, INCLUDE_NULL_VALUES";


nit: for other constants we dont prefix it with space (see the WITHOUT_ARRAY_WRAPPER) should probably stick to ensure we follow same format and there are not ghost spaces added.

This is actually needed similar to a FROM, else we will get error like '"Incorrect syntax near 'ASCFOR'."'

rohkhann · 2025-03-26T21:48:05Z

src/Core/Resolvers/DWSqlQueryBuilder.cs

+        /// </summary>
+        /// <param name="structure"></param>
+        /// <returns></returns>
+        private string BuildWithJsonFunc(SqlQueryStructure structure)


My understanding is this should exactly mimic the logic for mssql file, any chance of code reuse to prevent having to maintain two version os essentially the same TSQL code?

Valid point, I was thinking the same.

There are 3 differences comparing with mssql

dwsql does not support mutation, so there is no MultipleCreateOption available

dwsql never adopts the AddEscapeToLikeClauses

mssql can directly append the FOR JSON to their sub-query and use it for recursive calls, for dwsql, it has to be treated differently.

I've built a new base class BaseTSqlQueryBuilder for dwsql and mssql to better concise the code

src/Core/Resolvers/DWSqlQueryBuilder.cs

lingxiao-microsoft · 2025-03-27T00:56:52Z

src/Core/Resolvers/BaseSqlQueryBuilder.cs

+        /// </summary>
+        /// <param name="structure">Sql query structure to build query on</param>
+        /// <returns></returns>
+        protected virtual string BuildJsonPath(SqlQueryStructure structure)


Centralizing the functions so it can be used or override by both dwsql and mssql query builder.

lingxiao-microsoft · 2025-03-27T16:56:39Z

src/Core/Resolvers/BaseTSqlQueryBuilder.cs

+    /// Base query builder class for T-SQL engine
+    /// Can be used by dwsql and mssql
+    /// </summary>
+    public abstract class BaseTSqlQueryBuilder : BaseSqlQueryBuilder


I've built a new base class BaseTSqlQueryBuilder for dwsql and mssql to better concise the code

Aniruddh25 · 2025-03-27T23:53:17Z

src/Config/Converters/GraphQLRuntimeOptionsConverterFactory.cs

@@ -113,6 +113,17 @@ internal GraphQLRuntimeOptionsConverter(bool replaceEnvVar)
                                throw new JsonException($"Unexpected type of value entered for enable-aggregation: {reader.TokenType}");
                            }

+                            break;
+                        case "enable-dw-nto1joinopt":


based on our discussion, this may not be needed.

lingxiao-microsoft added 3 commits March 18, 2025 18:45

init commit

9ed5207

added unit tests

f84978f

minor updates

c5e59c3

lingxiao-microsoft changed the title ~~Use json_object to improve generated DW query performance~~ Improve N-1 join query performance for DW Mar 21, 2025

lingxiao-microsoft changed the title ~~Improve N-1 join query performance for DW~~ Improved N-1 join query performance for DW SQL Mar 21, 2025

lingxiao-microsoft added 2 commits March 24, 2025 11:41

Merge branch 'main' into lingxiao/dw-sql-perf

2a284e9

fixing format

1c74fa8

lingxiao-microsoft marked this pull request as ready for review March 24, 2025 23:10

lingxiao-microsoft requested review from Aniruddh25, aaronburtle, RubenCerna2079, ravishetye, rohkhann, neeraj-sharma2592, sourabh1007 and sezal98 as code owners March 24, 2025 23:10

vadeveka reviewed Mar 26, 2025

View reviewed changes

src/Core/Resolvers/DWSqlQueryBuilder.cs Show resolved Hide resolved

src/Core/Resolvers/DWSqlQueryBuilder.cs Show resolved Hide resolved

src/Core/Resolvers/DWSqlQueryBuilder.cs Show resolved Hide resolved

vadeveka reviewed Mar 26, 2025

View reviewed changes

rohkhann reviewed Mar 26, 2025

View reviewed changes

addressing comments

a484736

lingxiao-microsoft commented Mar 27, 2025

View reviewed changes

added a base class for mssql and dwsql

b423ea7

lingxiao-microsoft commented Mar 27, 2025

View reviewed changes

Aniruddh25 reviewed Mar 27, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved N-1 join query performance for DW SQL #2631

Improved N-1 join query performance for DW SQL #2631

lingxiao-microsoft commented Mar 21, 2025 •

edited

Loading

vadeveka Mar 26, 2025

lingxiao-microsoft Mar 26, 2025 •

edited

Loading

rohkhann Mar 26, 2025

rohkhann Mar 26, 2025

lingxiao-microsoft Mar 26, 2025 •

edited

Loading

vadeveka Mar 26, 2025

JerryNixon Mar 27, 2025 •

edited

Loading

Aniruddh25 Mar 27, 2025

rohkhann Mar 26, 2025

lingxiao-microsoft Mar 26, 2025

rohkhann Mar 26, 2025

lingxiao-microsoft Mar 26, 2025 •

edited

Loading

lingxiao-microsoft Mar 27, 2025 •

edited

Loading

lingxiao-microsoft Mar 27, 2025

Aniruddh25 Mar 27, 2025

Improved N-1 join query performance for DW SQL #2631

Are you sure you want to change the base?

Improved N-1 join query performance for DW SQL #2631

Conversation

lingxiao-microsoft commented Mar 21, 2025 • edited Loading

Why make this change?

What is this change?

How was this tested?

Manual Testing - Join Scenarios

Query 1-1 relation - As expected, optimization applied

Query N-1 relation - As expected, optimization applied

Query 1-N Relation - As expected, optimization not applied

Query N-N Relation - As expected, optimization not applied

Other Scenarios

Aggregation

Non-Join Query

Pagination

Choose a reason for hiding this comment

lingxiao-microsoft Mar 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lingxiao-microsoft Mar 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JerryNixon Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lingxiao-microsoft Mar 26, 2025 • edited Loading

Choose a reason for hiding this comment

lingxiao-microsoft Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lingxiao-microsoft commented Mar 21, 2025 •

edited

Loading

lingxiao-microsoft Mar 26, 2025 •

edited

Loading

lingxiao-microsoft Mar 26, 2025 •

edited

Loading

JerryNixon Mar 27, 2025 •

edited

Loading

lingxiao-microsoft Mar 26, 2025 •

edited

Loading

lingxiao-microsoft Mar 27, 2025 •

edited

Loading