Mutations on table with triggers - MsSql (#1630)

## Why make this change?
When there is a DML trigger configured on a table, we cannot use OUTPUT
clause to return data. When an insert DML trigger is configured, OUTPUT
clause cannot be used to return data because of the insert operation.
When an update DML trigger is configured, OUTPUT clause cannot be used
to return data because of the update operation. In such cases, we need
to use a subsequent select statement to get the data after the mutation.

## Details
**What would handling of different kind of mutations look like when a
trigger is enabled on a table?**

1. **DELETE:** We don't need to return any data for deletions, so
nothing will change for deletions.

2. **UPDATE:** For updates, we will cease to use the output clause.
Instead, we will do a subsequent select query on the table to get the
data. Since PK would already be provided to us we can easily do a select
query. Not to mention, we can do this only when we determine that there
is indeed a trigger enabled on the table. If not, we can continue with
the same query that we generate in present day.

3. **INSERT:** When it comes to insertions, we can perform insertions on
two kind of tables:
1. **Tables with non-auto generated primary keys:** When we have tables
with non-autogen PK, we can insert values of non-autogen PK fields into
a temporary table (we might not know value of every PK before hand
because PK can have a default value as well) and then perform an inner
join for the subsequent select on the actual table and the temporary
table based on the values of these PK fields.
2. **Tables with auto-gen primary keys:** Tables with an autogenerated
PK column fall into this category.. For tables with autogen PK, we would
not be aware of the PK beforehand and so we cannot directly perform the
subsequent select query. To get the autogen primary key, we can use
`SCOPE_IDENTITY()` method provided by Sql Server.. Once we get the
autogen PK, in addition to having an inner join for non-autogen PKs as
discussed in the previous approach, we can add an additional `WHERE`
predicate `where autogen_PK_field = SCOPE_IDENTITY()`. Let us discuss
more about this approach and the concerns:

**Delving deeper into
[Approach#3](#452 (comment)
We can use `OUTPUT` clause coupled with `INTO` clause to store the
values for the primary keys in a temporary table. And then do a
subsequent select query on the table to get the data using those primary
keys. In the temporary table, we insert only those columns from the PK
which are not autogenerated. Why? Because we cannot insert a value for
an autogenerated column even in a temporary table. Hence in the select
query, we will add an additional WHERE predicate (alongwith the join)
for the autogenerated column in the PK. It is to be noted that **MsSql
supports only one IDENTITY/autogenerated column per table**.
We can choose to go with this approach only if we determine that there
are any triggers enabled on the table.

Two assumptions which I am making here that I want to put ahead (btw, I
don't think those are valid use cases) are:

1. **What if primary key column has a datatype of timestamp?** This
concern arise because we cannot insert value for `timestamp` column
(which is what the approach includes - inserting value via `INTO`
clause). Well, `rowversion`/`timestamp` is intended for versioning and
tracking changes within a table, rather than serving as a primary key.
Also, by design, primary key values should not be modified after they
have been assigned to a record. So having a PK with timestamp datatype
does not seem to be a valid use case to me.

2. **What if the trigger updates the values of the primary keys** and
the subsequent select query we are going to perform returns no data
because the values of the PK has been changed in the table? This concern
arises because technically, it is possible for triggers to update the
value of PK columns. However, this is generally not recommended and can
lead to undesirable consequences. Changing the value of a primary key
can result in data integrity issues, as it may cause conflicts or break
referential integrity constraints if other tables are referencing the
primary key. It can also lead to confusion and difficulty in identifying
records uniquely.

## What is this change?

1. Added properties `IsUpdateDMLTriggerEnabled` and
`IsInsertDMLTriggerEnabled` to `SourceDefinition` class which if true
will indicate that there is a corresponding DML trigger enabled on the
table.
2. Added method `SqlMetadataProvider.PopulateTriggerMetadataForTable`
whose overridden implementation in `MsSqlMetadataProvider` will populate
the properties mentioned in (1) with appropriate metadata values.
3. Refactored how queries are built in the `SqlInsertStructure`,
`SqlUpsertQueryStructure`, `SqlUpdateStructure` so that when trigger is
configured for the mutation operation, a subsequent select query is used
instead of the `OUTPUT` clause to return data.
4. In case of insertions in table with auto-gen PK, we get the latest
value of the auto-gen column using `SCOPE_IDENTITY() ` function provided
by sql server. Reference:
https://learn.microsoft.com/en-us/sql/t-sql/functions/scope-identity-transact-sql?view=sql-server-ver16

## How was this tested?

- [x] Integration Tests - Added mutations tests to Rest/GQL performing
inserts/updates/upserts on tables with insert/update DML trigger
enabled. Tests are added for tables with autogen/non-autogen PK as the
queries differ in the two cases.

Author: ayush3797

config-generators/mssql-commands.txt

@@ -39,6 +39,8 @@ add InsertAndDisplayAllBooksUnderGivenPublisher --config "dab-config.MsSql.json"
 add GQLmappings --config "dab-config.MsSql.json" --source "GQLmappings" --permissions "anonymous:*" --rest true --graphql true
 add Bookmarks --config "dab-config.MsSql.json" --source "bookmarks" --permissions "anonymous:*" --rest true --graphql true
 add MappedBookmarks --config "dab-config.MsSql.json" --source "mappedbookmarks" --permissions "anonymous:*" --rest true --graphql true
+add FteData --config "dab-config.MsSql.json" --source fte_data --permissions "anonymous:*" --graphql "FteData:FteData"
+add InternData --config "dab-config.MsSql.json" --source intern_data --permissions "anonymous:*" --graphql "InternData:InternData"
 add BooksSold --config "dab-config.MsSql.json" --source "books_sold" --rest true --graphql "books_sold:books_sold" --permissions "anonymous:*"
 update GQLmappings --config "dab-config.MsSql.json" --map "__column1:column1,__column2:column2" --permissions "authenticated:*"
 update Publisher --config "dab-config.MsSql.json" --permissions "authenticated:create,read,update,delete" --rest true --graphql true --relationship books --target.entity Book --cardinality many

src/Config/DatabasePrimitives/DatabaseObject.cs

@@ -154,6 +154,21 @@ public class SourceDefinition
     public Dictionary<string, RelationshipMetadata> SourceEntityRelationshipMap { get; private set; } =
         new(StringComparer.InvariantCultureIgnoreCase);
 
+    /// <summary>
+    /// Indicates whether an update trigger enabled on the table.
+    /// The default value must be kept as false, meaning by default we assume no trigger is enabled.
+    /// Based on whether trigger is enabled, we use either OUTPUT (when no trigger is enabled) / or SELECT (when a trigger is enabled),
+    /// to return the data after a mutation operation.
+    /// </summary>
+    public bool IsUpdateDMLTriggerEnabled;
+
+    /// <summary>
+    /// Indicates whether an insert trigger enabled on the table.
+    /// The default value must be kept as false, meaning by default we assume no trigger is enabled.
+    /// Based on whether trigger is enabled, we use either OUTPUT (when no trigger is enabled) / or SELECT (when a trigger is enabled),
+    /// to return the data after a mutation operation.
+    public bool IsInsertDMLTriggerEnabled;
+
     /// <summary>
     /// Given the list of column names to check, evaluates
     /// if any of them is a nullable column when matched with the columns in this source definition.

src/Core/Resolvers/IQueryBuilder.cs

@@ -73,6 +73,12 @@ public interface IQueryBuilder
         /// an overridden implementation of the method.</exception>
         string BuildQueryToGetReadOnlyColumns(string schemaOrDatabaseParamName, string tableParamName);
 
+        /// <summary>
+        /// Builds the query to determine the number of enabled triggers on a database table.
+        /// Needed only for MsSql.
+        /// </summary>
+        string BuildFetchEnabledTriggersQuery() => throw new NotImplementedException();
+
         /// <summary>
         /// Adds database specific quotes to string identifier
         /// </summary>

src/Core/Resolvers/MsSqlQueryBuilder.cs

@@ -1,7 +1,12 @@
 // Copyright (c) Microsoft Corporation.
 // Licensed under the MIT License.
 
+using System.Data;
+using System.Text.Json;
+using System.Text.Json.Nodes;
+using Azure.DataApiBuilder.Config.DatabasePrimitives;
 using Azure.DataApiBuilder.Core.Configurations;
+using Azure.DataApiBuilder.Core.Models;
 using Azure.DataApiBuilder.Core.Resolvers;
 using Microsoft.Data.SqlClient;
 using Microsoft.Extensions.Logging;
@@ -40,5 +45,38 @@ public override Type SqlToCLRType(string sqlType)
         {
             return TypeHelper.GetSystemTypeFromSqlDbType(sqlType);
         }
+
+        /// <inheritdoc/>
+        public override async Task PopulateTriggerMetadataForTable(string entityName, string schemaName, string tableName, SourceDefinition sourceDefinition)
+        {
+            string enumerateEnabledTriggers = SqlQueryBuilder.BuildFetchEnabledTriggersQuery();
+            Dictionary<string, DbConnectionParam> parameters = new()
+            {
+                { $"{BaseQueryStructure.PARAM_NAME_PREFIX}param0", new(schemaName, DbType.String) },
+                { $"{BaseQueryStructure.PARAM_NAME_PREFIX}param1", new(tableName, DbType.String) }
+            };
+
+            JsonArray? resultArray = await QueryExecutor.ExecuteQueryAsync(
+                sqltext: enumerateEnabledTriggers,
+                parameters: parameters,
+                dataReaderHandler: QueryExecutor.GetJsonArrayAsync);
+            using JsonDocument sqlResult = JsonDocument.Parse(resultArray!.ToJsonString());
+
+            foreach (JsonElement element in sqlResult.RootElement.EnumerateArray())
+            {
+                string type_desc = element.GetProperty("type_desc").ToString();
+                if ("UPDATE".Equals(type_desc))
+                {
+                    sourceDefinition.IsUpdateDMLTriggerEnabled = true;
+                    _logger.LogInformation($"An update trigger is enabled for the entity: {entityName}");
+                }
+
+                if ("INSERT".Equals(type_desc))
+                {
+                    sourceDefinition.IsInsertDMLTriggerEnabled = true;
+                    _logger.LogInformation($"An insert trigger is enabled for the entity: {entityName}");
+                }
+            }
+        }
     }
 }

src/Core/Services/MetadataProviders/MsSqlMetadataProvider.cs

@@ -69,7 +69,7 @@ public abstract class SqlMetadataProvider<ConnectionT, DataAdapterT, CommandT> :
         public Dictionary<string, DatabaseObject> EntityToDatabaseObject { get; set; } =
             new(StringComparer.InvariantCulture);
 
-        private readonly ILogger<ISqlMetadataProvider> _logger;
+        protected readonly ILogger<ISqlMetadataProvider> _logger;
 
         public SqlMetadataProvider(
             RuntimeConfigProvider runtimeConfigProvider,
@@ -361,6 +361,19 @@ private async Task FillSchemaForStoredProcedureAsync(
         /// </summary>
         public abstract Type SqlToCLRType(string sqlType);
 
+        /// <summary>
+        /// Updates a table's SourceDefinition object's metadata with whether any enabled insert/update DML triggers exist for the table.
+        /// This method is only called for tables in MsSql.
+        /// </summary>
+        /// <param name="entityName">Name of the entity.</param>
+        /// <param name="schemaName">Name of the schema in which the table is present.</param>
+        /// <param name="tableName">Name of the table.</param>
+        /// <param name="sourceDefinition">Table definition to update.</param>
+        public virtual Task PopulateTriggerMetadataForTable(string entityName, string schemaName, string tableName, SourceDefinition sourceDefinition)
+        {
+            throw new NotImplementedException();
+        }
+
         /// <summary>
         /// Generates the map used to find a given entity based
         /// on the path that will be used for that entity.
@@ -975,10 +988,15 @@ private async Task PopulateSourceDefinitionAsync(
                        subStatusCode: DataApiBuilderException.SubStatusCodes.ErrorInInitialization);
             }
 
+            _entities.TryGetValue(entityName, out Entity? entity);
+            if (GetDatabaseType() is DatabaseType.MSSQL && entity is not null && entity.Source.Type is EntitySourceType.Table)
+            {
+                await PopulateTriggerMetadataForTable(entityName, schemaName, tableName, sourceDefinition);
+            }
+
             using DataTableReader reader = new(dataTable);
             DataTable schemaTable = reader.GetSchemaTable();
             RuntimeConfig runtimeConfig = _runtimeConfigProvider.GetConfig();
-            _entities.TryGetValue(entityName, out Entity? entity);
             foreach (DataRow columnInfoFromAdapter in schemaTable.Rows)
             {
                 string columnName = columnInfoFromAdapter["ColumnName"].ToString()!;

src/Core/Services/MetadataProviders/SqlMetadataProvider.cs

@@ -47,6 +47,8 @@ DROP TABLE IF EXISTS graphql_incompatible;
 DROP TABLE IF EXISTS GQLmappings;
 DROP TABLE IF EXISTS bookmarks;
 DROP TABLE IF EXISTS mappedbookmarks;
+DROP TABLE IF EXISTS fte_data;
+DROP TABLE IF EXISTS intern_data;
 DROP TABLE IF EXISTS books_sold;
 DROP SCHEMA IF EXISTS [foo];
 COMMIT;
@@ -256,6 +258,23 @@ CREATE TABLE mappedbookmarks
 	bkname nvarchar(50) NOT NULL
 )
 
+create Table fte_data(
+id int IDENTITY(5001,1),
+u_id int DEFAULT 2,
+name varchar(50),
+position varchar(20),
+salary int default 20,
+PRIMARY KEY(id, u_id)
+);
+
+create Table intern_data(
+id int,
+months int default 2 NOT NULL,
+name varchar(50),
+salary int default 15,
+PRIMARY KEY(id, months)
+);
+
 create table books_sold
 (
     id int PRIMARY KEY not null,
@@ -468,6 +487,13 @@ VALUES
 (10, 'Aaron', 'F.', 'Burtle', null, null)
 SET IDENTITY_INSERT authors_history OFF
 
+SET IDENTITY_INSERT fte_data ON
+insert into fte_data(id, name, position, salary) values(1, 'Ellie', 'Junior Dev', 20), (2, 'Chris', 'Senior Dev', 40);
+SET IDENTITY_INSERT fte_data OFF
+
+insert into intern_data(id, months, name) values(1, 3, 'Tess'), (2, 4, 'Frank');
+
+
 INSERT INTO revenues(id, category, revenue, accessible_role) VALUES (1, 'Book', 5000, 'Anonymous'), (2, 'Comics', 10000, 'Anonymous'),
 (3, 'Journals', 20000, 'Authenticated'), (4, 'Series', 40000, 'Authenticated');
 
@@ -545,3 +571,63 @@ EXEC('CREATE FUNCTION dbo.revenuesPredicate(@accessible_role varchar(20))
 -- Adding a security policy which would restrict access to the rows in revenues table for
 -- SELECT,UPDATE,DELETE operations using the filter predicate dbo.revenuesPredicate.
 EXEC('CREATE SECURITY POLICY dbo.revenuesSecPolicy ADD FILTER PREDICATE dbo.revenuesPredicate(accessible_role) ON dbo.revenues;');
+
+-- Create an after insert trigger for the fte_data table.
+-- This trigger will ensure that no employee record can be inserted with a salary of more than 100 and less than 0.
+EXEC('CREATE OR ALTER TRIGGER fte_data_after_insert_trigger ON fte_data AFTER INSERT AS
+BEGIN
+    UPDATE fte_data
+    SET fte_data.salary =
+	case
+	when fte_data.salary > 100 then 100
+    when fte_data.salary < 0 then 0
+	else fte_data.salary
+	end
+    FROM fte_data
+    INNER JOIN inserted ON fte_data.id = inserted.id AND fte_data.u_id = inserted.u_id;
+END;');
+
+-- Create an after update trigger for the fte_data table.
+-- This trigger will ensure that no employee record can be updated to have a salary of more than 150 and less than 0.
+EXEC('CREATE OR ALTER TRIGGER fte_data_after_update_trigger ON fte_data AFTER UPDATE AS
+BEGIN
+    UPDATE fte_data
+    SET fte_data.salary =
+	case
+	when fte_data.salary > 150 then 150
+    when fte_data.salary < 0 then 0
+	else fte_data.salary
+	end
+    FROM fte_data
+    INNER JOIN inserted ON fte_data.id = inserted.id AND fte_data.u_id = inserted.u_id;
+END;');
+
+-- Create an after insert trigger for the intern_data table.
+-- This trigger will ensure that no intern record can be inserted with a salary of more than 30 and less than 0.
+EXEC('CREATE OR ALTER TRIGGER intern_data_after_insert_trigger ON intern_data AFTER INSERT AS
+BEGIN
+    UPDATE intern_data
+    SET intern_data.salary =
+	case
+	when intern_data.salary > 30 then 30
+    when intern_data.salary < 0 then 0
+	else intern_data.salary
+	end
+    FROM intern_data
+    INNER JOIN inserted ON intern_data.id = inserted.id AND intern_data.months = inserted.months;
+END;');
+
+-- Create an after update trigger for the intern_data table.
+-- This trigger will ensure that no intern record can be updated to have a salary of more than 50 and less than 0.
+EXEC('CREATE OR ALTER TRIGGER intern_data_after_update_trigger ON intern_data AFTER UPDATE AS
+BEGIN
+    UPDATE intern_data
+    SET intern_data.salary =
+	case
+	when intern_data.salary > 50 then 50
+    when intern_data.salary < 0 then 0
+	else intern_data.salary
+	end
+    FROM intern_data
+    INNER JOIN inserted ON intern_data.id = inserted.id AND intern_data.months = inserted.months;
+END;');

src/Service.Tests/DatabaseSchema-MsSql.sql

@@ -2359,6 +2359,58 @@
         }
       }
     },
+    {
+      FteData: {
+        Source: {
+          Object: fte_data,
+          Type: Table
+        },
+        GraphQL: {
+          Singular: FteData,
+          Plural: FteData,
+          Enabled: true
+        },
+        Rest: {
+          Enabled: true
+        },
+        Permissions: [
+          {
+            Role: anonymous,
+            Actions: [
+              {
+                Action: *
+              }
+            ]
+          }
+        ]
+      }
+    },
+    {
+      InternData: {
+        Source: {
+          Object: intern_data,
+          Type: Table
+        },
+        GraphQL: {
+          Singular: InternData,
+          Plural: InternData,
+          Enabled: true
+        },
+        Rest: {
+          Enabled: true
+        },
+        Permissions: [
+          {
+            Role: anonymous,
+            Actions: [
+              {
+                Action: *
+              }
+            ]
+          }
+        ]
+      }
+    },
     {
       BooksSold: {
         Source: {