Fix the Serialization/Deserialization issue with $ prefix columns

[Alekhya-Polavarapu]

Why make this change?

Serialization and deserialization of metadata currently fail when column names are prefixed with the $ symbol.

Root cause

This issue occurs because we’ve enabled the ReferenceHandler flag in our System.Text.Json serialization settings. When this flag is active, the serializer treats $ as a reserved character used for special metadata (e.g., $id, $ref). As a result, any property name starting with $ is interpreted as metadata and cannot be deserialized properly.

What is this change?

This update introduces custom logic in the converter’s Write and Read methods to handle $-prefixed column names safely.

• During serialization, columns beginning with $ are escaped as "_$".

• During deserialization, this transformation is reversed to restore the original property names.

How was this tested

• Unit tests

[Copilot]

Pull Request Overview

This PR adds support for escaping dollar sign ($) prefixes in column names during JSON serialization/deserialization. This is necessary because JSON property names starting with $ have special meaning in some contexts (like MongoDB's JSON query language) and can cause serialization issues.

• Adds escape/unescape logic to DatabaseObjectConverter that converts $ to _$ during serialization and back during deserialization
• Updates InitializeObjects method to optionally generate column names with dollar sign prefixes
• Adds three new test methods to validate serialization/deserialization of database objects with dollar-prefixed column names

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 9 comments.

File	Description
src/Core/Services/MetadataProviders/Converters/DatabaseObjectConverter.cs	Implements escaping/unescaping logic for dollar sign prefixed column names in SourceDefinition properties
src/Service.Tests/UnitTests/SerializationDeserializationTests.cs	Adds test coverage for dollar sign handling and updates initialization method to support parameterized column name generation

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Alekhya-Polavarapu]

Can we confidently escape it to "_$", may be there might be columns that are genuinely created with _$.

[souvikghosh04]

$ or _$ are allowed in some databases but used in different ways. for e.g., in PostgreSQL, the column should be used in within double-quotes. In MS SQL, its allowed but in box brackets I suppose. MySQL allows too. its better to handle these edge cases and also test it with different databases.

[JerryNixon]

We should consider if this is an edge case we care about.

[Aniruddh25]

@Alekhya-Polavarapu - can you add the escaped char based on the database type, and just use such an escaped char that already is used by the respective db itself. So, in case it is \ for MSSQL, use that. and distinguish usage of \ by escaping it with 2 backslashes \\

[Aniruddh25]

Prompt what are the valid characters in a database... | Try in Copilot Chat
what are the valid characters in a database column name in Microsoft SQL Server?
Response
In Microsoft SQL Server, the rules for valid column names are:

---

✅ Allowed Characters

• Letters: A–Z and a–z
• Digits: 0–9 (but the name cannot start with a digit)
• Underscore: _
• At sign: @ (only for local variables, not recommended for column names)
• Hash: # (used for temporary tables, not typical for columns)

---

❌ Not Allowed

• Spaces (unless enclosed in delimiters)
• Special characters like !, ?, %, ^, &, *, (, ), +, =, /, \, |, ~
• Control characters or non-printable characters

---

Length

• Maximum: 128 characters

---

Delimiters for Exceptions

If you need spaces or reserved keywords:

• Use square brackets:

  [Column Name With Spaces]

• Or double quotes (if QUOTED_IDENTIFIER is ON):

  "Column Name With Spaces"

---

Best Practices

• Start with a letter or underscore
• Avoid reserved keywords (e.g., SELECT, TABLE)
• Stick to alphanumeric and underscore for portability

---

Do you want me to also include examples of valid and invalid column names for SQL Server? Or a quick T-SQL script to check if a name is valid?

[Aniruddh25]

For e.g. You can also use any of the not allowed characters as escape chars since that guarantees they wont be present in the column name.

[Copilot]

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Corrected spelling of 'deserilization' to 'deserialization'.

[Copilot]

The result of Equals() method call is not being asserted. This should be Assert.IsTrue(deserializedDatabaseTable.Equals(_databaseTable)) or similar to validate equality.

[Aniruddh25]

nit: name the argument whose value is true here

Suggested change

	InitializeObjects(true);
	InitializeObjects( generateDollaredColumn: true);