Update Regex named blocks from Unicode 4.0 to Unicode 17.0 #120623

Copilot · 2025-10-11T16:08:27Z

Summary

Successfully updated Regex named blocks from Unicode 4.0 to Unicode 17.0:

Added 59 new Unicode blocks including Arabic Extended-A/B, Balinese, Bamum, Cherokee Supplement, Cyrillic Extended-A/B/C, Devanagari Extended, Georgian Extended, Hangul Jamo Extended-A/B, Javanese, Kayah Li, Latin Extended-C/D/E, and many more
Maintained full backward compatibility by:
- Keeping IsGreek as alias for IsGreekandCoptic
- Restoring all previously removed blocks (IsCombiningMarksforSymbols, IsHighPrivateUseSurrogates, IsHighSurrogates, IsLowSurrogates, IsPrivateUse, IsPrivateUseArea)
- Maintaining correct alphabetical ordering
- Preserving original block values (e.g., IsSpecials)
- Maintaining test coverage for all blocks
Created GenRegexNamedBlocks tool for easy future Unicode updates with full documentation
Total named blocks: 167 (increased from 108)
No breaking changes - all existing block names remain available with correct values
Unicode 17.0: Verified that Unicode 17.0 has the same BMP blocks as 16.0, ensuring complete coverage of the latest Unicode standard
Updated Unicode update instructions: Added the GenRegexNamedBlocks tool to the official Unicode update process documentation in Updating-Unicode-Versions.md
Automated table generation: Moved s_propTable to a generated file (RegexCharClass.Tables.cs) that is automatically created by the GenRegexNamedBlocks tool, eliminating the need for manual updates
Added to all projects: Included RegexCharClass.Tables.cs in both the main library project and the source generator project

The changes enable .NET developers to use regular expressions with up-to-date Unicode block names while maintaining complete backward compatibility. Future Unicode updates will now be fully automated through the GenRegexNamedBlocks tool.

Fixes #118062

Original prompt

This section details on the original issue you should resolve

<issue_title>Consider updating named blocks in Regex</issue_title>
<issue_description>While testing for GB18030-2022 I noticed that Regex is missing multiple named blocks.
As per the documentation, the list is based on Unicode 4.0, while latest version is 16.0 https://www.unicode.org/charts/.

I think we already update Unicode categories. Updating named blocks is not as important as these can be easily defined as a range of characters manually.</issue_description>

Comments on the Issue (you are @copilot in this section)
@jeffhandley Marked as https://github.com/dotnet/runtime/labels/needs-further-triage for .NET 11. We should conclude during .NET 11 whether we want to start updating this regularly, or if there is a reason to keep it snapped to Unicode 4.0.

Fixes #118062

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

dotnet-policy-service · 2025-10-11T16:16:51Z

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Copilot

Pull Request Overview

This PR updates the Unicode named blocks in System.Text.RegularExpressions from Unicode 4.0 to Unicode 16.0, adding 52 new Unicode blocks while maintaining backward compatibility. The update enables regex patterns to match characters from modern scripts and symbol sets that were not previously supported.

Key changes include:

Added 52 new Unicode blocks covering various scripts like Arabic Extended, Balinese, Cherokee Supplement, and many others
Created a new tool GenRegexNamedBlocks to automate future Unicode updates
Updated test coverage to include all new blocks

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`src/libraries/System.Text.RegularExpressions/tools/GenRegexNamedBlocks/README.md`	Documentation for the new Unicode block generation tool
`src/libraries/System.Text.RegularExpressions/tools/GenRegexNamedBlocks/Program.cs`	Tool implementation for parsing Unicode Blocks.txt and generating regex named block entries
`src/libraries/System.Text.RegularExpressions/tools/GenRegexNamedBlocks/GenRegexNamedBlocks.csproj`	Project file for the Unicode block generation tool
`src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/RegexCharacterSetTests.cs`	Added test coverage for all 52 new Unicode blocks and removed tests for deprecated surrogate blocks
`src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexCharClass.cs`	Updated named blocks array with 52 new Unicode blocks, removed deprecated surrogate and private use blocks

src/libraries/System.Text.RegularExpressions/tools/GenRegexNamedBlocks/Program.cs

...ibraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexCharClass.cs

src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/RegexCharacterSetTests.cs

stephentoub · 2025-10-13T14:36:30Z

@copilot, can you update this for Unicode 17.0?
https://www.unicode.org/versions/Unicode17.0.0/

Copilot · 2025-10-13T15:08:09Z

@copilot, can you update this for Unicode 17.0?
https://www.unicode.org/versions/Unicode17.0.0/

Updated to Unicode 17.0. Unicode 17.0 has the same named blocks in the BMP (Basic Multilingual Plane) as Unicode 16.0, so no code changes were needed - only updated the documentation. Commit: e299e0c (will update with new hash).

stephentoub · 2025-10-15T16:02:19Z

@copilot, please address the build failures. It looks like you didn't add the new .cs file to the regex .csproj.

Copilot · 2025-10-15T16:21:18Z

@copilot, please address the build failures. It looks like you didn't add the new .cs file to the regex .csproj.

Fixed. Added RegexCharClass.Tables.cs to the csproj file. Commit: 0fa52a5

stephentoub · 2025-10-15T21:48:38Z

@copilot, the build is still failing:

Build FAILED.

D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\src\System\Text\RegularExpressions\RegexCharClass.cs(199,23): error CS0103: The name 's_propTable' does not exist in the current context [D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\gen\System.Text.RegularExpressions.Generator.csproj::TargetFramework=netstandard2.0]
D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\src\System\Text\RegularExpressions\RegexCharClass.cs(201,45): error CS0103: The name 's_propTable' does not exist in the current context [D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\gen\System.Text.RegularExpressions.Generator.csproj::TargetFramework=netstandard2.0]
D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\src\System\Text\RegularExpressions\RegexCharClass.cs(201,64): error CS0103: The name 's_propTable' does not exist in the current context [D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\gen\System.Text.RegularExpressions.Generator.csproj::TargetFramework=netstandard2.0]
D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\src\System\Text\RegularExpressions\RegexCharClass.cs(201,168): error CS0103: The name 's_propTable' does not exist in the current context [D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\gen\System.Text.RegularExpressions.Generator.csproj::TargetFramework=netstandard2.0]
D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\src\System\Text\RegularExpressions\RegexCharClass.cs(201,189): error CS0103: The name 's_propTable' does not exist in the current context [D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\gen\System.Text.RegularExpressions.Generator.csproj::TargetFramework=netstandard2.0]
D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\src\System\Text\RegularExpressions\RegexCharClass.cs(1809,23): error CS0103: The name 's_propTable' does not exist in the current context [D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\gen\System.Text.RegularExpressions.Generator.csproj::TargetFramework=netstandard2.0]
D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\src\System\Text\RegularExpressions\RegexCharClass.cs(1813,51): error CS0103: The name 's_propTable' does not exist in the current context [D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\gen\System.Text.RegularExpressions.Generator.csproj::TargetFramework=netstandard2.0]
D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\src\System\Text\RegularExpressions\RegexCharClass.cs(1824,34): error CS0103: The name 's_propTable' does not exist in the current context [D:\a\_work\1\s\src\libraries\System.Text.RegularExpressions\gen\System.Text.RegularExpressions.Generator.csproj::TargetFramework=netstandard2.0]
    0 Warning(s)
    8 Error(s)

Copilot · 2025-10-15T22:08:00Z

@copilot, the build is still failing:
Build FAILED.

...

Fixed. Added RegexCharClass.Tables.cs to the Generator project file as well. Commit: f3b8398

Added 52 new Unicode blocks and updated existing ones to match Unicode 16.0 specification. Total count increased from 108 to 160 named blocks. Co-authored-by: stephentoub <[email protected]>

- Add IsGreek as alias for IsGreekandCoptic for backward compatibility - Add comprehensive tests for 52 new Unicode blocks - Remove tests for deprecated surrogate and private use blocks - All 29,287 tests now passing Co-authored-by: stephentoub <[email protected]>

Document how to use the tool for future Unicode updates Co-authored-by: stephentoub <[email protected]>

Add back 6 blocks that were removed to avoid breaking changes: - IsCombiningMarksforSymbols (alias for IsCombiningDiacriticalMarksforSymbols) - IsHighPrivateUseSurrogates - IsHighSurrogates - IsLowSurrogates - IsPrivateUse (alias for IsPrivateUseArea) - IsPrivateUseArea Total blocks now: 167 (108 original + 59 new) Co-authored-by: stephentoub <[email protected]>

Add back test coverage for the 6 restored blocks: - IsHighSurrogates - IsHighPrivateUseSurrogates - IsLowSurrogates - IsPrivateUse Co-authored-by: stephentoub <[email protected]>

- Move IsHebrew to correct alphabetical position (before IsHighPrivateUseSurrogates) - Restore IsSpecials to original value "\uFFF0" (was incorrectly changed to "\uFFF0\u10000") Co-authored-by: stephentoub <[email protected]>

Unicode 17.0 has the same blocks in the BMP as Unicode 16.0, so no code changes needed. Updated documentation to reflect Unicode 17.0. Co-authored-by: stephentoub <[email protected]>

Remove trailing spaces on lines 17 and 37 to fix linting errors Co-authored-by: stephentoub <[email protected]>

Added step to run GenRegexNamedBlocks tool when updating Unicode version Co-authored-by: tarekgh <[email protected]>

- Made RegexCharClass partial (already was) - Created RegexCharClass.Tables.cs with s_propTable - Updated GenRegexNamedBlocks tool to generate complete .cs file - Tool now takes output file path parameter - Updated README with new tool usage Co-authored-by: stephentoub <[email protected]>

Include the new generated file in the csproj to fix build failures Co-authored-by: stephentoub <[email protected]>

Include the generated file in the Generator csproj to fix build failures for the source generator project Co-authored-by: stephentoub <[email protected]>

Copilot AI assigned Copilot and stephentoub Oct 11, 2025

Copilot started work on behalf of stephentoub October 11, 2025 16:08 View session

stephentoub added the area-System.Text.RegularExpressions label Oct 11, 2025

Copilot AI changed the title ~~[WIP] Update named blocks in Regex for Unicode 16.0~~ Update Regex named blocks from Unicode 4.0 to Unicode 16.0 Oct 11, 2025

Copilot AI requested a review from stephentoub October 11, 2025 17:01

Copilot finished work on behalf of stephentoub October 11, 2025 17:01

build-analysis bot mentioned this pull request Oct 11, 2025

Test failure: System.IO.Tests.FileInfo_SymbolicLinks.ResolveLinkTarget_Succeeds #120380

Open

stephentoub marked this pull request as ready for review October 12, 2025 01:17

stephentoub requested review from Copilot, joperezr and jozkee October 12, 2025 01:17

Copilot AI reviewed Oct 12, 2025

View reviewed changes

src/libraries/System.Text.RegularExpressions/tools/GenRegexNamedBlocks/Program.cs Show resolved Hide resolved

src/libraries/System.Text.RegularExpressions/tools/GenRegexNamedBlocks/Program.cs Show resolved Hide resolved

stephentoub reviewed Oct 12, 2025

View reviewed changes

...ibraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexCharClass.cs Show resolved Hide resolved

Copilot started work on behalf of stephentoub October 12, 2025 01:20 View session

Copilot finished work on behalf of stephentoub October 12, 2025 01:48

Copilot AI requested a review from stephentoub October 12, 2025 01:48

stephentoub reviewed Oct 12, 2025

View reviewed changes

Copilot started work on behalf of stephentoub October 12, 2025 02:15 View session

Copilot AI requested a review from stephentoub October 12, 2025 02:36

Copilot finished work on behalf of stephentoub October 12, 2025 02:36

Copilot started work on behalf of stephentoub October 12, 2025 02:40 View session

Copilot stopped work on behalf of stephentoub due to an error October 12, 2025 03:01
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

Copilot started work on behalf of stephentoub October 13, 2025 14:36 View session

Copilot AI changed the title ~~Update Regex named blocks from Unicode 4.0 to Unicode 16.0~~ Update Regex named blocks from Unicode 4.0 to Unicode 17.0 Oct 13, 2025

Copilot finished work on behalf of stephentoub October 13, 2025 15:09

build-analysis bot mentioned this pull request Oct 13, 2025

[browser] HalfTests.ExplicitConversion_FromSingle failing due to NaN != NaN #103347

Open

tarekgh added this to the 11.0.0 milestone Oct 15, 2025

build-analysis bot mentioned this pull request Oct 15, 2025

Intermittent build failure in AfterSourceBuild: "Could not write state file" #76488

Open

Copilot started work on behalf of stephentoub October 15, 2025 16:03 View session

Copilot stopped work on behalf of stephentoub due to an error October 15, 2025 16:22
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

build-analysis bot mentioned this pull request Oct 15, 2025

[android][arm64] System.Net.Sockets.Tests.SendTo_SyncForceNonBlocking.Datagram_UDP_ShouldImplicitlyBindLocalEndpoint fails with NetworkUnreachable #120526

Open

Copilot started work on behalf of stephentoub October 15, 2025 21:49 View session

Copilot stopped work on behalf of stephentoub due to an error October 15, 2025 22:08
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

stephentoub approved these changes Oct 15, 2025

View reviewed changes

stephentoub enabled auto-merge (squash) October 15, 2025 22:19

This was referenced Oct 16, 2025

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." dotnet/dnceng#1883

Open

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

Copilot AI and others added 13 commits October 16, 2025 22:29

Initial plan

77ac654

Update Regex named blocks from Unicode 4.0 to Unicode 16.0

9522c2b

Added 52 new Unicode blocks and updated existing ones to match Unicode 16.0 specification. Total count increased from 108 to 160 named blocks. Co-authored-by: stephentoub <[email protected]>

Add README for GenRegexNamedBlocks tool

2e06313

Document how to use the tool for future Unicode updates Co-authored-by: stephentoub <[email protected]>

Restore tests for surrogate and private use blocks

8b06b5a

Add back test coverage for the 6 restored blocks: - IsHighSurrogates - IsHighPrivateUseSurrogates - IsLowSurrogates - IsPrivateUse Co-authored-by: stephentoub <[email protected]>

Fix alphabetical order and IsSpecials value

5d70ee6

- Move IsHebrew to correct alphabetical position (before IsHighPrivateUseSurrogates) - Restore IsSpecials to original value "\uFFF0" (was incorrectly changed to "\uFFF0\u10000") Co-authored-by: stephentoub <[email protected]>

Update to Unicode 17.0

c867344

Unicode 17.0 has the same blocks in the BMP as Unicode 16.0, so no code changes needed. Updated documentation to reflect Unicode 17.0. Co-authored-by: stephentoub <[email protected]>

Fix trailing spaces in README.md

b7d0b8f

Remove trailing spaces on lines 17 and 37 to fix linting errors Co-authored-by: stephentoub <[email protected]>

Add GenRegexNamedBlocks tool to Unicode update instructions

41e2155

Added step to run GenRegexNamedBlocks tool when updating Unicode version Co-authored-by: tarekgh <[email protected]>

Add RegexCharClass.Tables.cs to project file

73527e8

Include the new generated file in the csproj to fix build failures Co-authored-by: stephentoub <[email protected]>

Add RegexCharClass.Tables.cs to Generator project

c2b5eec

Include the generated file in the Generator csproj to fix build failures for the source generator project Co-authored-by: stephentoub <[email protected]>

stephentoub force-pushed the copilot/update-named-blocks-in-regex branch from bae86bc to c2b5eec Compare October 17, 2025 02:35

stephentoub merged commit eed413d into main Oct 17, 2025
140 of 142 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update Regex named blocks from Unicode 4.0 to Unicode 17.0 #120623

Update Regex named blocks from Unicode 4.0 to Unicode 17.0 #120623

Copilot AI commented Oct 11, 2025 •

edited

Loading

Uh oh!

dotnet-policy-service bot commented Oct 11, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephentoub commented Oct 13, 2025

Uh oh!

Copilot AI commented Oct 13, 2025

Uh oh!

stephentoub commented Oct 15, 2025

Uh oh!

Copilot AI commented Oct 15, 2025

Uh oh!

stephentoub commented Oct 15, 2025

Uh oh!

Copilot AI commented Oct 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Update Regex named blocks from Unicode 4.0 to Unicode 17.0 #120623

Update Regex named blocks from Unicode 4.0 to Unicode 17.0 #120623

Conversation

Copilot AI commented Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Comments on the Issue (you are @copilot in this section)

Uh oh!

dotnet-policy-service bot commented Oct 11, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephentoub commented Oct 13, 2025

Uh oh!

Copilot AI commented Oct 13, 2025

Uh oh!

stephentoub commented Oct 15, 2025

Uh oh!

Copilot AI commented Oct 15, 2025

Uh oh!

stephentoub commented Oct 15, 2025

Uh oh!

Copilot AI commented Oct 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Oct 11, 2025 •

edited

Loading