Skip to content

[DO NOT MERGE] Switch to using darc vmr forwardflow #48351

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .config/dotnet-tools.json
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@
"isRoot": true,
"tools": {
"microsoft.dotnet.darc": {
"version": "1.1.0-beta.25209.3",
"version": "1.1.0-beta.25211.1",
"commands": [
"darc"
]
9 changes: 5 additions & 4 deletions eng/Version.Details.xml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<Dependencies>
<Source Mapping="sdk" Uri="https://github.com/dotnet/dotnet" Sha="88e8da66a7134e7339cc434af6be834203d5f255" BarId="263481" />
<ProductDependencies>
<Dependency Name="Microsoft.TemplateEngine.Abstractions" Version="10.0.100-preview.4.25210.1">
<Uri>https://github.com/dotnet/templating</Uri>
@@ -682,13 +683,13 @@
<Uri>https://github.com/dotnet/runtime</Uri>
<Sha>7618121e0f64098e2b81ed7c8cdf7024a1b87795</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.Darc" Version="1.1.0-beta.25209.3">
<Dependency Name="Microsoft.DotNet.Darc" Version="1.1.0-beta.25211.1">
<Uri>https://github.com/dotnet/arcade-services</Uri>
<Sha>98b747c01a9f431a5676a6ecf6ab2cf272ce69ee</Sha>
<Sha>8c5b42c0fb17f3bcb3b275c24fc99cea305fbef5</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.DarcLib" Version="1.1.0-beta.25209.3">
<Dependency Name="Microsoft.DotNet.DarcLib" Version="1.1.0-beta.25211.1">
<Uri>https://github.com/dotnet/arcade-services</Uri>
<Sha>98b747c01a9f431a5676a6ecf6ab2cf272ce69ee</Sha>
<Sha>8c5b42c0fb17f3bcb3b275c24fc99cea305fbef5</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.ScenarioTests.SdkTemplateTests" Version="10.0.0-preview.24602.1">
<Uri>https://github.com/dotnet/scenario-tests</Uri>
2 changes: 1 addition & 1 deletion eng/Versions.props
Original file line number Diff line number Diff line change
@@ -84,7 +84,7 @@
</PropertyGroup>
<PropertyGroup>
<!-- Dependency from https://github.com/dotnet/arcade-services -->
<MicrosoftDotNetDarcLibVersion>1.1.0-beta.25209.3</MicrosoftDotNetDarcLibVersion>
<MicrosoftDotNetDarcLibVersion>1.1.0-beta.25211.1</MicrosoftDotNetDarcLibVersion>
</PropertyGroup>
<PropertyGroup>
<!-- Dependency from https://github.com/dotnet/aspire -->
20 changes: 7 additions & 13 deletions eng/pipelines/templates/steps/vmr-pull-updates.yml
Original file line number Diff line number Diff line change
@@ -43,16 +43,12 @@ steps:
workingDirectory: ${{ parameters.vmrPath }}

- script: |
./eng/vmr-sync.sh \
--vmr ${{ parameters.vmrPath }} \
--tmp $(Agent.TempDirectory) \
--azdev-pat '$(dn-bot-all-orgs-code-r)' \
--branch ${{ parameters.vmrBranch }} \
--repository "sdk:${{ parameters.targetRef }}" \
--recursive \
--remote "sdk:$(Agent.BuildDirectory)/sdk" \
--tpn-template $(Agent.BuildDirectory)/sdk/src/VirtualMonoRepo/THIRD-PARTY-NOTICES.template.txt \
--ci \
./eng/vmr-sync.sh \
--vmr ${{ parameters.vmrPath }} \
--tmp $(Agent.TempDirectory) \
--azdev-pat '$(dn-bot-all-orgs-code-r)' \
--branch ${{ parameters.vmrBranch }} \
--ci \
--debug

if [ "$?" -ne 0 ]; then
@@ -70,14 +66,12 @@ steps:
condition: eq(variables['Agent.OS'], 'Windows_NT')

- powershell: |
cp .\src\VirtualMonoRepo\source-mappings.json '${{ parameters.vmrPath }}'
./eng/vmr-sync.ps1 `
-vmr ${{ parameters.vmrPath }} `
-tmp $(Agent.TempDirectory) `
-azdevPat '$(dn-bot-all-orgs-code-r)' `
-branch ${{ parameters.vmrBranch }} `
-repository "sdk:${{ parameters.targetRef }}" `
-recursive `
-tpnTemplate $(Agent.BuildDirectory)/sdk/src/VirtualMonoRepo/THIRD-PARTY-NOTICES.template.txt `
-ci `
-debugOutput

93 changes: 13 additions & 80 deletions eng/vmr-sync.ps1
Original file line number Diff line number Diff line change
@@ -1,57 +1,26 @@
<#
.SYNOPSIS

This script is used for synchronizing the dotnet/dotnet VMR locally. This means pulling new
code from various repositories into the 'dotnet/dotnet' repository.
This script is used for synchronizing the current repository into a local VMR.
It pulls the current repository's code into the specified VMR directory for local testing or
Source-Build validation.

.DESCRIPTION

The script is used during CI to ingest new code based on dotnet/sdk but it can also help
for reproducing potential failures during sdk's PRs, namely to fix the Source-Build.
Another usecase is to try manually synchronizing a given commit of some repo into the VMR and
trying to Source-Build the VMR. This can help when fixing the Source-Build but using a commit
from a not-yet merged branch (or fork) to test the fix will help.

The tooling that synchronizes the VMR will need to clone the various repositories into a temporary
folder. These clones can be re-used in future synchronizations so it is advised you dedicate a
folder to this to speed up your re-runs.
The tooling used for synchronization will clone the VMR repository into a temporary folder if
it does not already exist. These clones can be reused in future synchronizations, so it is
recommended to dedicate a folder for this to speed up re-runs.

.EXAMPLE
Synchronize current sdk and all dependencies into a local VMR:
Synchronize current repository into a local VMR:
./vmr-sync.ps1 -vmrDir "$HOME/repos/dotnet" -tmpDir "$HOME/repos/tmp"

Synchronize the VMR to a specific commit of dotnet/runtime using custom fork:
./vmr-sync.ps1 `
-repository runtime:e7e71da303af8dc97df99b098f21f526398c3943 `
-remote runtime:https://github.com/yourfork/runtime `
-tmpDir "$HOME/repos/tmp"

.PARAMETER tmpDir
Required. Path to the temporary folder where repositories will be cloned

.PARAMETER vmrBranch
Optional. Branch of the 'dotnet/dotnet' repo to synchronize. The VMR will be checked out to this branch

.PARAMETER recursive
Optional. Recursively synchronize all the source build dependencies (declared in Version.Details.xml)
This is used when performing the full synchronization during sdk's CI and the final VMR sync.
Defaults to false unless no repository is supplied in which case a recursive sync of sdk is performed.

.PARAMETER remote
Optional. Additional remote to use during the synchronization
This can be used to synchronize to a commit from a fork of the repository
Example: 'runtime:https://github.com/yourfork/runtime'

.PARAMETER repository
Optional. Repository + git ref separated by colon to synchronize to.
This can be a specific commit, branch, tag.
If not supplied, the revision of the parent sdk repository of this script will be used (recursively).
Example: 'runtime:my-branch-name'

.PARAMETER tpnTemplate
Optional. Template for the header of VMRs THIRD-PARTY-NOTICES file.
Defaults to src/VirtualMonoRepo/THIRD-PARTY-NOTICES.template.txt

.PARAMETER azdevPat
Optional. Azure DevOps PAT to use for cloning private repositories.

@@ -68,10 +37,7 @@ param (
[Parameter(Mandatory=$true, HelpMessage="Path to the temporary folder where repositories will be cloned")]
[string][Alias('t', 'tmp')]$tmpDir,
[string][Alias('b', 'branch')]$vmrBranch,
[switch]$recursive,
[string]$remote,
[string][Alias('r')]$repository,
[string]$tpnTemplate = "src/VirtualMonoRepo/THIRD-PARTY-NOTICES.template.txt",
[string]$azdevPat,
[string][Alias('v', 'vmr')]$vmrDir,
[switch]$ci,
@@ -88,23 +54,13 @@ function Highlight {
Write-Host "> $($args[0])" -ForegroundColor 'Cyan'
}

$sdkDir = (Split-Path -Parent $scriptRoot)

# If sdk is a repo, we're in an sdk and not in the dotnet/dotnet repo
if (Test-Path -Path "$sdkDir/.git" -PathType Container) {
$additionalRemotes = "sdk:$sdkDir"
}

if ($remote) {
$additionalRemotes = "$additionalRemotes $remote"
}

$verbosity = 'verbose'
if ($debugOutput) {
$verbosity = 'debug'
}
# Validation

$sdkDir = (Split-Path -Parent $scriptRoot)
if (-not (Test-Path -Path $sdkDir -PathType Container)) {
Fail "Directory '$sdkDir' does not exist. Please specify the path to the dotnet/sdk repo"
exit 1
@@ -115,19 +71,8 @@ if (-not $tmpDir) {
exit 1
}

if (-not (Test-Path -Path $tpnTemplate -PathType Leaf)) {
Fail "File '$tpnTemplate' does not exist. Please specify a valid path to the THIRD-PARTY-NOTICES template"
exit 1
}

# Sanitize the input

# Default when no repository is provided
if (-not $repository) {
$repository = "sdk:$(git -C $sdkDir rev-parse HEAD)"
$recursive = $true
}

if (-not $vmrDir) {
$vmrDir = Join-Path $tmpDir 'dotnet'
}
@@ -169,32 +114,20 @@ $dotnetRoot = InitializeDotNetCli -install:$true
$dotnet = "$dotnetRoot\dotnet.exe"
& "$dotnet" tool restore

Highlight "Starting the synchronization of '$repository'.."
Highlight "Starting the synchronization of VMR.."

# Synchronize the VMR
$darcArgs = (
"darc", "vmr", "update",
"--vmr", $vmrDir,
"darc", "vmr", "forwardflow",
$vmrDir,
"--tmp", $tmpDir,
"--$verbosity",
"--tpn-template", $tpnTemplate,
"--discard-patches",
"--generate-credscansuppressions",
$repository
"--$verbosity"
)

if ($recursive) {
$darcArgs += ("--recursive")
}

if ($ci) {
$darcArgs += ("--ci")
}

if ($additionalRemotes) {
$darcArgs += ("--additional-remotes", $additionalRemotes)
}

if ($azdevPat) {
$darcArgs += ("--azdev-pat", $azdevPat)
}
@@ -205,7 +138,7 @@ if ($LASTEXITCODE -eq 0) {
Highlight "Synchronization succeeded"
}
else {
Fail "Synchronization of dotnet/dotnet to '$repository' failed!"
Fail "Synchronization of repo to VMR failed!"
Fail "'$vmrDir' is left in its last state (re-run of this script will reset it)."
Fail "Please inspect the logs which contain path to the failing patch file (use -debugOutput to get all the details)."
Fail "Once you make changes to the conflicting VMR patch, commit it locally and re-run this script."
Loading