Preserve file encoding in file-based app source files

[Copilot]

• Understand the issue: SourceFile.Save() uses Encoding.UTF8 which adds BOM, breaking shebang lines
• Locate the problematic code in FileLevelDirectiveHelpers.cs
• Identify test infrastructure in FileBasedAppSourceEditorTests.cs
• Fix SourceFile struct to preserve encoding using SourceText.Encoding property
• Add test case to verify BOM is not added when file doesn't have one
• Add test case to verify BOM is preserved when file has one
• Build and run new tests to validate the fix (tests pass)
• Address PR feedback (round 1):
  • Use stackalloc for 3-byte BOM detection buffer (with netstandard2.0 fallback)
  • Leverage SourceText's Encoding property instead of custom tracking
  • Update tests to use _testAssetsManager.CreateTestDirectory() infrastructure
  • Add issue reference to test documentation
• Address PR feedback (round 2):
  • Move <see href/> inside <summary> elements
  • Simplify to use SourceText's built-in encoding auto-detection (pass encoding: null)
  • Remove custom BOM detection logic entirely
• Address PR feedback (round 3):
  • Add test for non-UTF-8 encoding preservation (UTF-16)
  • Verify that SourceText auto-detection works for encodings beyond UTF-8
• Address PR feedback (round 4):
  • Use collection expressions in assertions (is pattern matching)
  • Verify complete file content in PreservesNoBomEncoding test
• Address PR feedback (round 5):
  • Fix test to use explicit newlines instead of raw string literal
  • Resolve test failure on Windows due to line ending normalization
• Manually verify fix works end-to-end with shebang scripts and UTF-16 files
• Reply to all PR comments

Original prompt

---

This section details on the original issue you should resolve

<issue_title>file-based apps: dotnet add file.cs package changes encoding of file.cs</issue_title>
<issue_description>### Describe the bug
The SDK changes encoding of file-based app files when adding packages.

To Reproduce

1 - Create a a file.cs (UTF-8 encoded)
2 - Add a shebang : #!/usr/bin/env dotnet run and chmod +x file.cs
3- Run it: ./file.cs
4 - Add any package dotnet add file.cs AnyNugetPackageIdHere => file.cs is converted to UTF-8 with BOM
5 - Running ./file.cs again after the package is added results in failure:

> ./file.cs
./file.cs: line 1: #!/usr/bin/env: No such file or directory

Further technical details

.NET 10 SDK, 1.0.100</issue_description>

<agent_instructions>Look at the root cause analysis and suggested fix in the issue comments</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@DamianEdwards Analysis from Copilot:

Root Cause Analysis

The issue is in the SourceFile struct in [src/Cli/Microsoft.DotNet.FileBasedPrograms/FileLevelDirectiveHelpers. cs](https://github.com/dotnet/sdk/blob/e300dea82e0248b0801f706914c96073c1bd5350/src/Cli/Microsoft.DotNet. FileBasedPrograms/FileLevelDirectiveHelpers.cs#L256-L274):

public void Save()
{
    using var stream = File.Open(Path, FileMode.Create, FileAccess. Write);
    using var writer = new StreamWriter(stream, Encoding.UTF8);
    Text.Write(writer);
}

Problem: Encoding. UTF8 (the static property) emits a UTF-8 BOM by default. When dotnet add file. cs package <package> modifies the file and saves it, the BOM (0xEF 0xBB 0xBF) is prepended to the file content.

On Unix-like systems, the shebang #!/usr/bin/env dotnet run becomes <BOM>#!/usr/bin/env dotnet run, which the kernel doesn't recognize as a valid interpreter directive, causing:

./file.cs: line 1: #!/usr/bin/env: No such file or directory

Fix: The Save() method should preserve the original file's encoding/BOM characteristics. Specifically, for files without a BOM, it should use:

new UTF8Encoding(encoderShouldEmitUTF8Identifier: false)

The Load() method should also detect and store whether the original file had a BOM so it can be preserved on save. </comment_new>

• Fixes file-based apps: dotnet add file.cs package <package> changes encoding of file.cs #52054

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[jjonescz]

@copilot

[jjonescz]

@copilot

[Copilot]

Pull request overview

This PR fixes a critical bug where dotnet add file.cs package <package> was inadvertently adding a UTF-8 BOM to file-based app source files, breaking shebang (#!) scripts on Unix-like systems. The fix preserves the original encoding state (with or without BOM) by leveraging Roslyn's SourceText encoding detection and preservation capabilities.

Key Changes:

• Modified SourceFile.Load() to auto-detect encoding by passing encoding: null to SourceText.From()
• Updated SourceFile.Save() to use the detected encoding from SourceText.Encoding property
• Added comprehensive tests verifying BOM preservation for both cases (with and without BOM)

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/Cli/Microsoft.DotNet.FileBasedPrograms/FileLevelDirectiveHelpers.cs	Updated SourceFile.Load() to auto-detect encoding and SourceFile.Save() to preserve it using SourceText.Encoding
test/dotnet.Tests/CommandTests/Run/FileBasedAppSourceEditorTests.cs	Added two test cases: PreservesNoBomEncoding() for files without BOM and PreservesBomEncoding() for files with BOM

[333fred]

A few minor comments on the tests.

[jjonescz]

@333fred @RikkiGibson for another look, thanks

[RikkiGibson]

LGTM, comment is not blocking.

[jjonescz]

/backport to release/10.0.2xx

[github-actions]

Started backporting to release/10.0.2xx (link to workflow run)