Rework description in CHANGES.txt for GH #4717. #4720
Conversation
Changes: * Partially restore restore original GH SCons#4717 CHANGES.txt description * Change MSVS to MSVC in RELEASE.txt * Remove extraneous two blank lines introduced in common.py when variable moved
|
Notes for updated text PR. Observations:
The runtimes with powershell 5 are an order of magnitude slower than with powershell 7 in Visual Studio 2022 VCPKG
Example SConstruct times with Powershell 5 (runner results sorted low to high in secs):
Example SConstruct times with Powershell 7 (runner results sorted low to high in secs):
|
|
Windows runner experiments measuring only elapsed time for subprocess invocation of msvc batch files. Notes:
Windows Runner ExperimentsPowershell 5 and 7 results in seconds:
|
Windows VMWare VM ExperimentsPowershell 5 and 7 results in seconds:
VM State Reverted Between Experiments
VM State Not Reverted Between Experiments
|
jcbrill
changed the title
|
Does the CHANGES.txt blurb make sense to anyone else? |
|
Makes sense, yes. I might be proposing a bit of a rewording which would affect tone and tense, but not content - if I get to it, I promise to be careful not to lose any of the message. I'm currently proofreading some Python docu PRs on request and the energy may not extend to this one. We'll see what @bdbaddog thinks. |
|
@mwichmann I starting to wonder if the implementation should be adaptive to the os environment rather than hard-coded. PS 7 is now hard-coded before PS 5 in the msvc environment. The telemetry calls are currently hard-coded to use PS 5 in the batch files which is why PS 5 was added to the msvc environment. The optional vcpkg batch files will use PS 7 or 5 based on which is found first on the os system path. The SCons behavior is only the same as the msvc batch files if PS 7 precedes PS 5 on the os system path. It does in the windows runner but it may not in every end user's system configuration. I'm wondering if the order of PS 7 and 5 in the msvc environment should be based in the order on the user's system path. Similarly, should There are 3 known PS 7 path locations and 2 known PS 5 path locations in the PSModulePath in This is a situation where invoking the msvc batch files in the minimal msvc environment could be different than when running in the os environment. Any thoughts? [Always] @REM Send Telemetry if user's VS is opted-in
if "%VSCMD_SKIP_SENDTELEMETRY%"=="" (
if "%VSCMD_DEBUG%" NEQ "" (
@echo [DEBUG:%~nx0] Sending telemetry
powershell.exe -NoProfile -Command "& {Import-Module '%~dp0\Microsoft.VisualStudio.DevShell.dll'; Send-VsDevShellTelemetry -NewInstanceType Cmd;}"
) else (
START "" /B powershell.exe -NoProfile -Command "& {if($PSVersionTable.PSVersion.Major -ge 3){Import-Module '%~dp0\Microsoft.VisualStudio.DevShell.dll'; Send-VsDevShellTelemetry -NewInstanceType Cmd; }}" > NUL
)
)[If Installed] :: Call powershell which may or may not invoke bootstrap if there's a version mismatch
SET Z_POWERSHELL_EXE=
FOR %%i IN (pwsh.exe powershell.exe) DO (
IF EXIST "%%~$PATH:i" (
SET "Z_POWERSHELL_EXE=%%~$PATH:i"
GOTO :gotpwsh
)
)IMPORTANT: Use of diff format in code fragments intended as easy method of highlighting rather than as actual additions and subtractions.
os.environ[PATH]=
+ C:\Program Files\PowerShell\7;
C:\Program Files\MongoDB\Server\5.0\bin;
C:\aliyun-cli;
C:\vcpkg;
C:\Program Files (x86)\NSIS\;
C:\tools\zstd;
C:\Program Files\Mercurial\;
C:\hostedtoolcache\windows\stack\3.5.1\x64;
C:\cabal\bin;
C:\\ghcup\bin;
C:\mingw64\bin;
C:\Program Files\dotnet;
C:\Program Files\MySQL\MySQL Server 8.0\bin;
C:\Program Files\R\R-4.4.2\bin\x64;
C:\SeleniumWebDrivers\GeckoDriver;
C:\SeleniumWebDrivers\EdgeDriver\;
C:\SeleniumWebDrivers\ChromeDriver;
C:\Program Files (x86)\sbt\bin;
C:\Program Files (x86)\GitHub CLI;
C:\Program Files\Git\bin;
C:\Program Files (x86)\pipx_bin;
C:\npm\prefix;
C:\hostedtoolcache\windows\go\1.24.2\x64\bin;
C:\hostedtoolcache\windows\Python\3.9.13\x64\Scripts;
C:\hostedtoolcache\windows\Python\3.9.13\x64;
C:\hostedtoolcache\windows\Ruby\3.0.7\x64\bin;
C:\Program Files\OpenSSL\bin;
C:\tools\kotlinc\bin;
C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.452-9\x64\bin;
C:\Program Files\ImageMagick-7.1.1-Q16-HDRI;
C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin;
C:\ProgramData\kind;
C:\ProgramData\Chocolatey\bin;
C:\Windows\system32;
C:\Windows;
C:\Windows\System32\Wbem;
+ C:\Windows\System32\WindowsPowerShell\v1.0\;
C:\Windows\System32\OpenSSH\;
C:\Program Files\dotnet\;
+ C:\Program Files\PowerShell\7\;
C:\Program Files\Microsoft\Web Platform Installer\;
C:\Program Files\TortoiseSVN\bin;
C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;
C:\Program Files\Microsoft SQL Server\150\Tools\Binn\;
C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;
C:\Program Files (x86)\WiX Toolset v3.14\bin;
C:\Program Files\Microsoft SQL Server\130\DTS\Binn\;
C:\Program Files\Microsoft SQL Server\140\DTS\Binn\;
C:\Program Files\Microsoft SQL Server\150\DTS\Binn\;
C:\Program Files\Microsoft SQL Server\160\DTS\Binn\;
C:\Strawberry\c\bin;
C:\Strawberry\perl\site\bin;
C:\Strawberry\perl\bin;
C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin;
C:\Program Files\CMake\bin;
C:\ProgramData\chocolatey\lib\maven\apache-maven-3.9.9\bin;
C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;
C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;
C:\Program Files\nodejs\;
C:\Program Files\Git\cmd;
C:\Program Files\Git\mingw64\bin;
C:\Program Files\Git\usr\bin;
C:\Program Files\GitHub CLI\;
c:\tools\php;
C:\Program Files (x86)\sbt\bin;
C:\Program Files\Amazon\AWSCLIV2\;
C:\Program Files\Amazon\SessionManagerPlugin\bin\;
C:\Program Files\Amazon\AWSSAMCLI\bin\;
C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;
C:\Program Files\LLVM\bin;
C:\Users\runneradmin\.dotnet\tools;
C:\Users\runneradmin\.cargo\bin;
C:\Users\runneradmin\AppData\Local\Microsoft\WindowsApps
os.environ[PSMODULEPATH]=
+ C:\Users\runneradmin\Documents\PowerShell\Modules;
+ C:\Program Files\PowerShell\Modules;
+ c:\program files\powershell\7\Modules;
C:\\Modules\az_12.4.0;
- C:\Users\packer\Documents\WindowsPowerShell\Modules;
+ C:\Program Files\WindowsPowerShell\Modules;
+ C:\Windows\system32\WindowsPowerShell\v1.0\Modules;
C:\Program Files\Microsoft SQL Server\130\Tools\PowerShell\Modules\
os.environ[USERPROFILE]=
C:\Users\runneradmin |
|
adaptive: possibly a good thing. the runners are weird overloaded environments "with everything" that probably don't match dev boxes. is the ps version chosen purely by First In PATH, or are there other ways to select? |
In the optional vcpkg.bat call chain, the first In vsdevcmd.bat call chain, Both versions of powershell will use It may not be the worst idea to re-evaluate what is added to the For example, powershell 5 has been in the system path and Powershell 7 is not installed by default. Windows EnvironmentsThe Platform/win32.py environment is crazy minimal for modern Windows: As mentioned previously ad nauseum, the default Platform/win32 and msvc environments lack the common Windows system environment variables: The following system environment variables are known to be referenced in the msvc 2022 batch files:
If one were to compare the paths generated from the command-line versus from SCons, one would find minor differences in the paths configured. To date, either it doesn't matter and/or users adjust the paths themselves after the fact. It is likely that anyone attempting to run It almost appears like the windows environments were written before windows supported 64-bit platforms and really never evolved. Legend:
Default Windows 11, 10: Path=
C:\WINDOWS\system32;
- C:\WINDOWS;
+ C:\WINDOWS\System32\Wbem;
+ C:\WINDOWS\System32\WindowsPowerShell\v1.0\;
- C:\WINDOWS\System32\OpenSSH\;
- C:\Users\%USERNAME%\AppData\Local\Microsoft\WindowsApps;
- PSModulePath=
- C:\Program Files\WindowsPowerShell\Modules;
- C:\WINDOWS\system32\WindowsPowerShell\v1.0\ModulesDefault Windows 8, 7, Vista: Path=
C:\Windows\system32;
- C:\Windows;
+ C:\Windows\System32\Wbem;
+ C:\Windows\System32\WindowsPowerShell\v1.0\
- PSModulePath=
- C:\Windows\system32\WindowsPowerShell\v1.0\Modules\Default Windows XP: Path=
C:\WINDOWS\system32;
- C:\WINDOWS;
+ C:\WINDOWS\System32\Wbem |
|
@mwichmann #4226 appears to avoid all of the above as |
|
Based on what you say, I wonder if using a restricted environment for the setup run is the right choice. After all, we do want to accurately detect what's available. Then, of course, we'd have to figure out how many more things have to be exported to the build-time |
|
Looking at the vcpkg stuff, I guess it's this stanza? :: Call powershell which may or may not invoke bootstrap if there's a version mismatch
SET Z_POWERSHELL_EXE=
FOR %%i IN (pwsh.exe powershell.exe) DO (
IF EXIST "%%~$PATH:i" (
SET "Z_POWERSHELL_EXE=%%~$PATH:i"
GOTO :gotpwsh
)
)
:gotpwsh
"%Z_POWERSHELL_EXE%" -NoProfile -ExecutionPolicy Unrestricted -Command "iex (get-content \"%~dfp0\" -raw)#"Since the temporary variable with the powershell location is unset immediately afterward and not further used. And the expression evaluation seems to just be path fiddling? So how does this manage to go off in the weeds and take a long time? Do I even want to know (well, "want" == No here, but anyway...) |
I suspect if that PR were to be pushed forward, someone (probably me) would have suggested changing it to use |
|
All good questions. Blaise Pascal (Letter 16, 1657):
Apologies for the verbose responses that follow.
That is the dilemma in a nutshell. This is of course anathema to purists that do not want any possibility of tight coupling to an end user's environments. On the other hand, running in an OS-like environment probably removes a considerable maintenance/support burden. For preserved variables, the What a user cannot do right now is set the environment used to invoke the msvc batch files. Using the os.environ would eliminate the need to import variables into the environment and manually set the system path elements. With the VS VCPKG component, we would likely have to While this is less of problem today, keep in mind that the command shell environment could be 32-bit, and/or the python architecture could be 32-bit on a Windows 64-bit host. Running the environment set by running the vcvars batch file would be subtly different than running the same batch files from a 64-bit shell and 64-bit build of python. This effects the system variables. I once used a Windows third-party toolbar to launch a command shell. The tool was 32-bit so the default launched environment was configured for 32-bit applications. I believe that the vcvars batch files should be run in the same environment native to the host. This would require more fiddling and explicitly launching the vcvars batch files via a fully qualified path to the command interpreter due to WoW redirection. On ARM64, there is a subtle difference in vcvars results between running an amd64 build of python versus an ARM64 build of python due to PROCESSOR_ARCHITECTURE. The latest toolsets for VS2022 have native ARM64 builds. While unlikely, on ARM64, python could be 32-bit arm, 32-bit x86, 64-bit amd64, or 64-bit ARM64. I believe there are 3 shell environment (I could be wrong): 32-bit x86, 64-bit amd64, or 64-bit ARM64. It is almost trivial to run the one-time vcvars batch files with an explicit command interpreter if necessary.
Bit of trivia: the Passing env=dict(os.environ) to
My guess is that when calling itself as a powershell script, there is delay is introduced in one or more of the following: determining the default PSModulePath, loading powershell modules, downloading and "installing" information from github. Somewhere in the snippet below (alas some code removed) lies the root cause of the delay. Rough call sequence:
|
|
Forgot to add: an SCons |
Not sure I quite get this comment. If you open a "developer command prompt" (cmd or ps variant) you get a shell with a boatload of environment variables set. Of course that's different to what we run the build commands with, given the scons policy we all know and love. That doesn't seem likely to have been your point? |
I did a poor job trying to provide a reminder that in the current code, the environment passed to the subprocess invocation of the msvc batch files ignores everything in the user's MSCommon/common.py MSCommon/vc.py The get_output invocation from MSCommon/vc.py is not passed an env argument. Inside get_output, a new environment is materialized if the env argument is None. It does not matter what is in the user's env['ENV']. It won't be used at all. In this respect, the msvc tool initialization may be very different than other tools. That is why suggestions like the following aren't going to work: The environment in which the msvc batch files is invoked won't contain the |
RELEASE.txt
Outdated
| @@ -35,7 +35,7 @@ CHANGED/ENHANCED EXISTING FUNCTIONALITY | |||
| one from PEP 308 introduced in Python 2.5 (2006). The idiom being | |||
| replaced (using and/or) is regarded as error prone. | |||
|
|
|||
| - MSVS: The default Windows powershell 7 path is added before the default | |||
| - MSVC: The default Windows powershell 7 path is added before the default | |||
There was a problem hiding this comment.
Needs a blurb about the user impact. Perhaps something like.
"In some environments (such as Github Actions) this can yield a significant reduction in the time to initialize MSVC. (Without this change you may see > 10x slowdown) "
replace ##x slowdown with your best approximation of your test results.
The idea is that anyone reading this looking for a version with a fix should know from this blurb that it is indeed fixed.
RELEASE.txt is what gets sent to the mailing lists and other notification channels. Users then would have to go to the CHANGES.txt.
So RELEASE.txt should have more or less the same content as this releases CHANGE.txt blurb, but organized differently, without the attribution.
|
Prototype "adaptive" powershell results. Changes:
Results from two environments are shown below:
I think this is way it should be done. The only hesitation is in including the Current User powershell paths. In this limited use context, I don't think it matters and would be similar to running in the OS environment. Note: Not all of the PSModulePath elements are passed to the msvc environment. Only the elements that comprise the powershell "default paths" if PSModulePath is undefined. This does "fix" the runtime issue in the windows runner. Prototype ResultsDiff code language used for highlighting purposes. Windows runner (PS 7 & PS 5): os.environ[PATH]=
+ C:\Program Files\PowerShell\7;
C:\Program Files\MongoDB\Server\5.0\bin;
C:\aliyun-cli;
C:\vcpkg;
C:\Program Files (x86)\NSIS\;
C:\tools\zstd;
C:\Program Files\Mercurial\;
C:\hostedtoolcache\windows\stack\3.5.1\x64;
C:\cabal\bin;
C:\\ghcup\bin;
C:\mingw64\bin;
C:\Program Files\dotnet;
C:\Program Files\MySQL\MySQL Server 8.0\bin;
C:\Program Files\R\R-4.4.2\bin\x64;
C:\SeleniumWebDrivers\GeckoDriver;
C:\SeleniumWebDrivers\EdgeDriver\;
C:\SeleniumWebDrivers\ChromeDriver;
C:\Program Files (x86)\sbt\bin;
C:\Program Files (x86)\GitHub CLI;
C:\Program Files\Git\bin;
C:\Program Files (x86)\pipx_bin;
C:\npm\prefix;
C:\hostedtoolcache\windows\go\1.24.3\x64\bin;
C:\hostedtoolcache\windows\Python\3.9.13\x64\Scripts;
C:\hostedtoolcache\windows\Python\3.9.13\x64;
C:\hostedtoolcache\windows\Ruby\3.0.7\x64\bin;
C:\Program Files\OpenSSL\bin;
C:\tools\kotlinc\bin;
C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.452-9\x64\bin;
C:\Program Files\ImageMagick-7.1.1-Q16-HDRI;
C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin;
C:\ProgramData\kind;
C:\ProgramData\Chocolatey\bin;
C:\Windows\system32;
C:\Windows;
C:\Windows\System32\Wbem;
+ C:\Windows\System32\WindowsPowerShell\v1.0\;
C:\Windows\System32\OpenSSH\;
C:\Program Files\dotnet\;
+ C:\Program Files\PowerShell\7\;
C:\Program Files\Microsoft\Web Platform Installer\;
C:\Program Files\TortoiseSVN\bin;
C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;
C:\Program Files\Microsoft SQL Server\150\Tools\Binn\;
C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;
C:\Program Files (x86)\WiX Toolset v3.14\bin;
C:\Program Files\Microsoft SQL Server\130\DTS\Binn\;
C:\Program Files\Microsoft SQL Server\140\DTS\Binn\;
C:\Program Files\Microsoft SQL Server\150\DTS\Binn\;
C:\Program Files\Microsoft SQL Server\160\DTS\Binn\;
C:\Strawberry\c\bin;
C:\Strawberry\perl\site\bin;
C:\Strawberry\perl\bin;
C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin;
C:\Program Files\CMake\bin;
C:\ProgramData\chocolatey\lib\maven\apache-maven-3.9.9\bin;
C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;
C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;
C:\Program Files\nodejs\;
C:\Program Files\Git\cmd;
C:\Program Files\Git\mingw64\bin;
C:\Program Files\Git\usr\bin;
C:\Program Files\GitHub CLI\;
c:\tools\php;
C:\Program Files (x86)\sbt\bin;
C:\Program Files\Amazon\AWSCLIV2\;
C:\Program Files\Amazon\SessionManagerPlugin\bin\;
C:\Program Files\Amazon\AWSSAMCLI\bin\;
C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;
C:\Program Files\LLVM\bin;
C:\Users\%USERNAME%\.dotnet\tools;
C:\Users\%USERNAME%\.cargo\bin;
C:\Users\%USERNAME%\AppData\Local\Microsoft\WindowsApps
os.environ[PSMODULEPATH]=
+ C:\Users\%USERNAME%\Documents\PowerShell\Modules;
+ C:\Program Files\PowerShell\Modules;
+ c:\program files\powershell\7\Modules;
C:\\Modules\az_12.4.0;
- C:\Users\packer\Documents\WindowsPowerShell\Modules;
+ C:\Program Files\WindowsPowerShell\Modules;
+ C:\Windows\system32\WindowsPowerShell\v1.0\Modules;
C:\Program Files\Microsoft SQL Server\130\Tools\PowerShell\Modules\
env[PATH]=
C:\Windows\System32;
C:\Windows\System32\Wbem;
+ C:\Program Files\PowerShell\7;
C:\Windows\System32\WindowsPowerShell\v1.0\
+ env[PSModulePath]=
+ C:\Users\%USERNAME%\Documents\PowerShell\Modules;
+ C:\Program Files\PowerShell\Modules;
+ C:\Program Files\PowerShell\7\Modules;
+ C:\Program Files\WindowsPowerShell\Modules;
+ C:\Windows\System32\WindowsPowerShell\v1.0\ModulesLocal machine (PS 5): os.environ[PATH]=
C:\Data\venv\scons-dev\3.11.9\Scripts;
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64_win\compiler;
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\ia32_win\compiler;
C:\Windows\system32;
C:\Windows;
C:\Windows\System32\Wbem;
+ C:\Windows\System32\WindowsPowerShell\v1.0\;
C:\Windows\System32\OpenSSH\;
C:\Program Files\dotnet\;
C:\Software\PuTTY\;
C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;
C:\Program Files\Microsoft SQL Server\120\Tools\Binn\;
C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.0\;
C:\Users\%USERNAME%\.dnx\bin;
C:\Program Files\Microsoft DNX\Dnvm\;
C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;
C:\Program Files\Microsoft\Web Platform Installer\;
C:\Software\TortoiseHg\;
C:\Software\Git\cmd;
C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;
C:\Software\TortoiseGit\bin;
C:\Users\%USERNAME%\.pyenv\pyenv-win\bin;
C:\Users\%USERNAME%\AppData\Local\Microsoft\WindowsApps;
C:\Users\%USERNAME%\.dotnet\tools;
C:\Users\%USERNAME%\AppData\Local\GitHubDesktop\bin;
C:\Users\%USERNAME%\AppData\Roaming\npm;
C:\Users\%USERNAME%\AppData\Local\JpegminiPro4\;
C:\Users\%USERNAME%\AppData\Local\Markdown Monster
os.environ[PSMODULEPATH]=
+ C:\Program Files\WindowsPowerShell\Modules;
+ C:\Windows\system32\WindowsPowerShell\v1.0\Modules
env[PATH]=
C:\Windows\System32;
C:\Windows\System32\Wbem;
C:\Windows\System32\WindowsPowerShell\v1.0\
+ env[PSModulePath]=
+ C:\Program Files\WindowsPowerShell\Modules;
+ C:\Windows\System32\WindowsPowerShell\v1.0\Modules |
|
Adaptive powershell path elements and PSModulePath handling in MSCommon/common.py branch of this PR: If desired, this PR could be updated with the changes from the branch. Commit message:
Additional known "challenges" with the existing MSCommon/common and MSCommon/vc code should probably be documented. Comments and questions welcome. PS: I am time challenged for the next 7 days. |
|
The vcpkg search for the powershell executable will use pwsh.exe (PS 7) if it is anywhere on the path; otherwise, it will use powershell.exe (PS 5) if it is on the path. The path order does not matter: PS 7 will be used even if the PS 5 path precedes the PS 7 path. Sigh. |
|
That's what I see in the code snip above - it's a loop which exits on first match. |
|
Yep. The logic in the commit above is still the way to go. I have updated and committed MSCommon/common.py in the other branch with a revised comment: jcbrill@43270cb |
Corrected answer. vcvars Telemetry:
VS vcpkg powershell executable priority (path order does not matter):
|
|
So given how I now understand this information, here are two possible rewordings of the changelog snip - one verbose (I have this tendency too :-) ), and one very concise in a just-the-facts-without-details form. Just tossing these out there, feel free to ignore these, or meld them into a new version, or whatever.
|
Isn't |
|
yes, you're right. I thought I had read it had flipped, but -latest does still point to 2022. |
|
New theory under investigation: a non-trivial component of the slowdown might be the first time powershell.exe is invoked and not necessarily the VS vcpkg initialization. The VS vcpkg initialization could trigger the first powershell invocation depending on the value of VSCMD_SKIP_SENDTELEMETRY value. |
I get re-confused every go-round of this. So the old powershell causes problems, but that's the situation we've always had? I do see that standalone vcpkg has been installed on the runner since at least the 2019 image, but only 2022 and 2025 have the vcpkg VS component |
Yes. Old powershell without PSModulePath (important). I believe that the first invocation of powershell creates a command cache. Subsequent invocations may be faster. To date, all of my tests have VSCMD_SKIP_SENDTELEMETRY=1, therefore: I may run some tests without VSCMD_SKIP_SENDTELEMETRY to see if there is a runtime penalty in 2019 as well. It is possible there may have been a runtime penalty in all environments that has gone undetected until now. For example, if it takes a long time to build your project, 30 seconds might go unnoticed. Unfortunately, these types of issues could lead to the conclusion that "SCons is slow" mostly due to the restricted msvc environment in which the msvc commands are run. Running a powershell.exe command to list the available modules ( The list of available modules is very long. I have a few more experiments to try that may shed some light on the differences between PS 5 with and without PSModulePath and PS 7 with and without PSModulePath. This of course could be a red herring.
The aggressively minimal msvc environment is [was] the issue:
I'm still working on the differences between with and without PSModulePath for powershell.exe and pwsh.exe. My guess is that the idea is to use the latest and greatest if possible. |
|
a thought... apparently, a famous powershell slowdown is Optimizing (.net) Assemblies after a .net update. if it doesn't have its path, no chance of picking up ones that are already done in the image? |
I don't know. I believe the first time powershell/pwsh is invoked it may build a cache of available commands. Under certain circumstances, this is really expensive for powershell.exe and not as bad as with pwsh.exe. I'm not sure what building the cache entails. It appears like when PSModulePath is not defined, both powershell.exe and pwsh.exe retrieve the registry value of PSModulePath. However, powershell.exe appears to ignore the PS7 path components. This is not the same as defining PSModulePath to the OS value. With VSCMD_SKIP_SENDTELEMETRY defined, the VS vcpkg initialization will be the first powershell.exe/pwsh.exe invocation. I'm leaning towards there is a penalty experienced due to building the powershell module cache. Putting finishing touches on some experiments to show what may be happening. If it is the initial call, it is kind of a Heisenberg challenge to prove. |
|
I believe there are two initialization penalties:
The magnitude of the penalty for 1 is based on how the restricted msvc environment is configured. The magnitude of the penalty for 2 is based on how the restricted msvc enironment is configured and external factors. It may be the case that both 1 and 2 are experienced on the first VS vcpkg batch file invocation. To test this theory, the powershell executables are called using the restricted msvc environment. The command executed is The powershell.exe and/or pwsh.exe executables are invoked three (3) times consecutively. The msvc vcvars batch file is then called three (3) times consecutively. The first invocation of powershell compared to subsequent invocations appears to imply an initialization "setup cost". The first invocation of the msvc vcvars batch files appears to imply an intialization "setup cost" due to vcpkg initializing. The magnitude of the elapsed times can be wildly inconsistent. For example, in Run 1 below, the first powershell invocation took 74.23 seconds. Twenty minutes earlier I think it tool 34 seconds in one experiment. For traditional SCons builds, the powershell and vcpkg initialization costs are hidden within the msvc batch file invocation time. For the experiments below, the powershell first time cost is outside of the msvc batch file invocation time leaving only the vcpkg first time cost. Run 1 below indicates that even after the initial powershell cost, all powershell invocations are "expensive" compared to the other configurations. Run 1 is how the restricted msvc environment was configured prior to adding the PS 7 executable path. When PSModulePath is undefined, the powershell executables appear to inspect the PSModulePath definition in the registry. Run 1 - windows 2022, powershell.exe only, PSModulePath undefineos.environ[PSMODULEPATH]=
C:\Users\runneradmin\Documents\PowerShell\Modules
C:\Program Files\PowerShell\Modules
c:\program files\powershell\7\Modules
C:\\Modules\az_12.4.0
C:\Users\packer\Documents\WindowsPowerShell\Modules
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules
C:\Program Files\Microsoft SQL Server\130\Tools\PowerShell\Modules\
CONFIG = PATH_5_MODULEPATH_NA
env[PATH]=
C:\Windows\System32;
C:\Windows\System32\Wbem;
C:\Windows\System32\WindowsPowerShell\v1.0\
env[PSModulePath]=
CALLING='powershell.exe gci env: | Out-String -Width 4096'
stdout:PSMODULEPATH=
C:\Users\runneradmin\Documents\WindowsPowerShell\Modules
C:\\Modules\az_12.4.0
C:\Users\packer\Documents\WindowsPowerShell\Modules
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules
C:\Program Files\Microsoft SQL Server\130\Tools\PowerShell\Modules\
ELAPSED_TIME=74.23, cmd='powershell.exe gci env: | Out-String -Width 4096'
ELAPSED_TIME=20.51, cmd='powershell.exe gci env: | Out-String -Width 4096'
ELAPSED_TIME=20.39, cmd='powershell.exe gci env: | Out-String -Width 4096'
ELAPSED_TIME=24.95, vc_script='C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Auxiliary\\Build\\vcvars64.bat'
ELAPSED_TIME=20.97, vc_script='C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Auxiliary\\Build\\vcvars64.bat'
ELAPSED_TIME=22.44, vc_script='C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Auxiliary\\Build\\vcvars64.bat'Run 2 - windows 2022, powershell.exe only, OS PSModulePathos.environ[PSMODULEPATH]=
C:\Users\runneradmin\Documents\PowerShell\Modules
C:\Program Files\PowerShell\Modules
c:\program files\powershell\7\Modules
C:\\Modules\az_12.4.0
C:\Users\packer\Documents\WindowsPowerShell\Modules
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules
C:\Program Files\Microsoft SQL Server\130\Tools\PowerShell\Modules\
CONFIG = PATH_5_MODULEPATH_OS
env[PATH]=
C:\Windows\System32
C:\Windows\System32\Wbem
C:\Windows\System32\WindowsPowerShell\v1.0\
env[PSModulePath]=
C:\Users\runneradmin\Documents\PowerShell\Modules
C:\Program Files\PowerShell\Modules
c:\program files\powershell\7\Modules
C:\\Modules\az_12.4.0
C:\Users\packer\Documents\WindowsPowerShell\Modules
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules
C:\Program Files\Microsoft SQL Server\130\Tools\PowerShell\Modules\
CALLING='powershell.exe gci env: | Out-String -Width 4096'
stdout:PSModulePath
C:\Users\runneradmin\Documents\PowerShell\Modules
C:\Program Files\PowerShell\Modules
c:\program files\powershell\7\Modules
C:\\Modules\az_12.4.0
C:\Users\packer\Documents\WindowsPowerShell\Modules
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules
C:\Program Files\Microsoft SQL Server\130\Tools\PowerShell\Modules\
ELAPSED_TIME=0.39, cmd='powershell.exe gci env: | Out-String -Width 4096'
ELAPSED_TIME=0.29, cmd='powershell.exe gci env: | Out-String -Width 4096'
ELAPSED_TIME=0.30, cmd='powershell.exe gci env: | Out-String -Width 4096'
ELAPSED_TIME=5.22, vc_script='C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Auxiliary\\Build\\vcvars64.bat'
ELAPSED_TIME=1.02, vc_script='C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Auxiliary\\Build\\vcvars64.bat'
ELAPSED_TIME=0.98, vc_script='C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Auxiliary\\Build\\vcvars64.bat'Run 3 - windows 2022, pwsh.exe and powershell.exe, PSModulePath = known PS 7 and PS 5 module pathsos.environ[PSMODULEPATH]=
C:\Users\runneradmin\Documents\PowerShell\Modules
C:\Program Files\PowerShell\Modules
c:\program files\powershell\7\Modules
C:\\Modules\az_12.4.0
C:\Users\packer\Documents\WindowsPowerShell\Modules
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules
C:\Program Files\Microsoft SQL Server\130\Tools\PowerShell\Modules\
CONFIG = PATH_7_MODULEPATH_75
env[PATH]=
C:\Windows\System32
C:\Windows\System32\Wbem
C:\Program Files\PowerShell\7
C:\Windows\System32\WindowsPowerShell\v1.0\
env[PSModulePath]=
C:\Users\runneradmin\Documents\PowerShell\Modules
C:\Program Files\PowerShell\Modules
C:\Program Files\PowerShell\7\Modules
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules
CALLING='pwsh.exe -Command "gci env: | Out-String -Width 4096"'
stdout:PSModulePath
C:\Users\runneradmin\Documents\PowerShell\Modules
C:\Program Files\PowerShell\Modules
C:\Program Files\PowerShell\7\Modules
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules
ELAPSED_TIME=0.91, cmd='pwsh.exe -Command "gci env: | Out-String -Width 4096"'
ELAPSED_TIME=0.44, cmd='pwsh.exe -Command "gci env: | Out-String -Width 4096"'
ELAPSED_TIME=0.44, cmd='pwsh.exe -Command "gci env: | Out-String -Width 4096"'
CALLING='powershell.exe gci env: | Out-String -Width 4096'
stdout:PSModulePath
C:\Users\runneradmin\Documents\PowerShell\Modules
C:\Program Files\PowerShell\Modules
C:\Program Files\PowerShell\7\Modules
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules
ELAPSED_TIME=0.33, cmd='powershell.exe gci env: | Out-String -Width 4096'
ELAPSED_TIME=0.31, cmd='powershell.exe gci env: | Out-String -Width 4096'
ELAPSED_TIME=0.31, cmd='powershell.exe gci env: | Out-String -Width 4096'
ELAPSED_TIME=3.22, vc_script='C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Auxiliary\\Build\\vcvars64.bat'
ELAPSED_TIME=1.19, vc_script='C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Auxiliary\\Build\\vcvars64.bat'
ELAPSED_TIME=1.20, vc_script='C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Auxiliary\\Build\\vcvars64.bat' |
|
Maybe the PS 5 assemblies are not optimized? Powershell.exe is invoked when VSCMD_SKIP_SENDTELEMETRY is undefined. This could possibly incur the powershell setup cost, if any, before vcpkg initialization. It could appear in 2019 if that were the case although the magnitude could be very different. |
|
The .net assembly thing is probably something different, but it does involve populating something called a "Native Image Cache". In a StackOverflow post, someone had reported seeing first-time startup of around a minute, which sounds oddly familiar. I snipped this attempted explanation from "the net":
It's probably different because the cache isn't in a powershell-specific place, so the PS path fiddling should not affect it. The paths are |
…eCleanup, and PSModuleAnalysisCachePath into the limited msvc environment when defined. Environment variables added to msvc environment import list: * POWERSHELL_TELEMETRY_OPTOUT: opt-out of telemetry. * PSDisableModuleAnalysisCacheCleanup: disable checking for modules that no longer exist when writing module analysis cache. * PSModuleAnalysisCachePath: path and filename used to cache data about modules and their cmdlets. The PSModuleAnalysisCachePath is defined in windows runners 2019, 2022, and 2025. The significant delays experienced in the 2022 and 2025 runners when running the msvc batch files due to vcpkg with powershell 5 and without PSModulePath defined appear to be eliminated when the powershell module analysis cache path is propagated to the msvc environment. Additional changes: * Move the PS7 path after the PS5 path. * Add the elapsed execution time of the msvc batch files to the debug log.
|
@mwichmann Windows runner 2022 and 2025 root cause of delays likely identified. Experimentation keeps pointing to the powershell module analysis cache as likely having a significant impact. The environment variable The significant delays experienced in the 2022 and 2025 runners when running the msvc batch files due to The powershell module analysis cache appears to be pre-populated in the windows runners (which makes sense). Running PS 5 with the module analysis cache location The impact of Additional environment variables propagated to the msvc environment when defined:
The PS 7 executable path was moved after the PS 5 executable path. The text in CHANGES.txt and RELEASE.txt still need to be updated. |
|
Sounds good. I haven't changed my opinion: keeping environment variables strictly contained in SCons is a recipe for problems when they come from a complex and carefully curated system (such as the Microsoft compiler suite - more than just C++). By the way (nitpicking), I've recently learned that we probably shouldn't call it "PowerShell 7", it's PowerShell Core, the (supposedly) open-sourced version that rebased onto .NET core - that's able to run cross-platform. That actually started with v6, has now moved on to v7, |
I agree in principle. The limited msvc environment was carefully crafted 20 years ago. Not so much today. Unfortunately, using the OS environment comes with its own challenges. I believe the msvc batch files preserve the OS definitions of On the other hand, it may be easier to maintain an "exclude" list rather than an "include" list.
Fair enough. Referring to the powershell executable versions which are approximately 5.1(?) and 7.5(?) seemed easier than referring to powershell.exe vs pwsh.exe directly. |
Of course. 20+ years ago the developers didn't put in this scheme for no reason, I'm not saying that. The risk of bad settings in a developer's environment, that produce unreliable results, or non-reproducible ones ("nobody else has the same settings") are very real. I just re-ran the Intel OneAPI compiler suite's setup on Linux, which, like msvc (and maybe worse), hunts around for the setup scripts of installed components, and in the end it added 42 new environment variables as well as prepending eight path segments to the front of |
I was attempting to say that the limited msvc environment has not kept up with the changes introduced in successive Windows versions (e.g., 2000, 7, 8, 10, 11, etc). The burr in my saddle is not passing along all of the Windows system default variables. That would not have solved the powershell issue though. One could argue that powershell (5.X) has been a Windows system program for a long time and the optional environment variables should be supported (especially those in common with Powershell Core). There are two other "issues" with the current SCons processing of the msvc environment environment output:
SCons does not keep all of the environment variables that the msvc batch files create. I'm not sure keeping them is problematic. It might even make debugging users issues easier as the environment could be dumped. The simplest way of keeping new variables is a difference of the set of environment variables coming out of the batch files and the set of those that went into the environment. One could keep all of the new variables. The environment variables in both environments could be compared to see if they changed and if so, process the changed variables. This gets trickier depending on the meaning of the environment variable and whether or not a change should be preserved. Deletions could be handled as well. |
|
@mwichmann For your amusement... This is the entirety of the MSVC 6.0 VC 6.0 was used in anger well past VS 2005. |
Is this only in the case that vcpkg is installed? |
|
I'm not sure what you are asking. The delays were caused by the configuration of the limited msvc environment and the execution of a powershell script that likely involves loading additional commands/modules which in turn involves the powershell module analysis cache. Installing vcpkg causes a powershell script to be run. The limited msvc environment causes the significant delays with SCons 4.9.1 and the windows 2022 and 2025 runners. vcpkg initialization appears to trigger powershell initialization delays with the module analysis cache. The previous PR added pwsh.exe to the msvc environment path which appears to eliminate the significant delays. Including The Prior to the previous PR which added pwsh.exe to the path, any powershell.exe command or script invocation that required loading additional powershell commands/modules in the limited msvc environment without Running this command Without The first time of 53.09 seconds typically ranges from 30-120 seconds. Subsequent invocations are typically around 20 seconds. With This shows there is a cost without passing the existing There is an initial cost and subsequent cost for powershell.exe without the module analysis cache. There is an initial cost to initialize vcpkg that appears to be larger than subsequent invocations. vcpkg likely causes powershell.exe to load additional modules for the first time. Note: powershell.exe is used to initialize sending telemetry data. In all windows runners, simply loading a DLL does not appear to cause a delay. This is why the runner 2019 runs were as expected. This PR:
|
|
I was curious if the slowdown only happens when vcpkg has been installed. Also of note, a google finds that we could set a reasonable default for PSModuleAnalysisCachePath without copying the shell env. Above also gives (seems to at least) definitive answers about powershell's user of shell env variables. |
I'd expect we want to be sure to get the one that's already been initialized, or we don't benefit from the cache warming effect. That may well turn out to be the default for that PS version - but if it was, then why do we see the problem with the variable unset (I have only PS 5 installed, and my cache is in |
For the msvc batch file invocations in the windows 2022 and 2025 runners: yes. I don't know if there are any additional uninstalled extra VS components that rely on powershell that could have materialized similar behavior. The powershell.exe invocation to setup telemetry in the windows 2019/2022/2035 runners does not manifest a significant slowdown.
From earlier comments:
In the windows 2019, 2022, and 2025 windows runners, The reason for importing It may not make any sense to define our own location in the environment as powershell will use its own default location when The benefit for the windows runners is that the powershell analysis module cache appears to be populated before the msvc batch files are invoked. Even if the cache were not pre-prepopulated, there may be a good reason why a user would like the cache to be populated in a certain location. For example, in the windows runners, if the cache were moved to a different drive (e.g., D:) there might be a runtime benefit. Perhaps a user is manually saving and restoring the powershell module analysis cache from their own known location. The powershell environment configuration could be
That was the source, and contains the rationale, for adding Some amount of caution needs to be exercised in assuming all environments will behave like the windows 2019/2022/2025 runners. For the limited msvc environment, importing At present, Future work might consider what was described in the alternate branch mentioned above:
P.S.: A similar argument could be made for importing |
|
@mwichmann I was in the process of writing the above and committing before seeing your post. Please check the following for accuracy.
This.
There is a cache persistence difference between running on a physical pc or a virtual machine environment compared to the windows runners. On a physical pc or a virtual machine, the default cache once constructed (if at all) is persistent. To emulate the windows runner behavior in a virtual machine, one would have to "reset the vm state" after each run. The default locations of the module analysis cache when
For the following, the size of the pre-populated cache in the Windows 2022 runner is 1,610,258 bytes (~1.6MB) as reported by python's
The only way to guarantee the same behavior as the shell environment is to: 1) add pwsh.exe to the msvc environment path, 2) include The prior PR does 1. This PR does 1 and 2. A limited form of 3 was discussed in my previous post under "future work".
It is confusing.
That would be my guess. PS 7 (Core) does not appear to add the hex suffix to the end of the |
|
@mwichmann Research implementation windows runner logs showing the complete OS environment and limited msvc environment, and the powershell module analysis cache file sizes (0 if non-existent). Let me know if you have any questions. These logs are representative of the windows runner findings to date. The logs also provide a decent view of the windows runner configuration. The table below contains modified versions of the windows 2022 logs for four experiments with the pure python implementation. The windows runner timestamp prefix was removed for readability. This is using the research implementation and not the SCons implementation. Log records legend:
PS 5.x and No Cache: SCons 4.9.1 and earlier. PS 7.x and No Cache: Previous PR (i.e., current SCons master). PS 7.x and Cache: This PR. PS 5.x and Cache: This PR (implied if pwsh.exe not on path). |
|
I think that Adding a shell environment variable to the limited msvc environment does not mean it will be preserved in the caller's SCons environment. Other than Effectively, powershell running in the limited msvc environment would be roughly equivalent to running powershell in the shell environment subject to the minimal This same argument applies for adding known Windows system variables (e.g., The diagram below shows the shell environment members of the limited environment in which the msvc batch files are invoked and the environment variables that are preserved in the caller's SCons environment. flowchart TD;
A[
**Construct Limited MSVC Environment**<br>
*==SCons/Platform/win32.py==*<br/>
PATH [minimal]<sup>1</sup>
PATHEXT [minimal]<sup>2</sup>
SystemDrive
SystemRoot
TEMP
TMP
USERPROFILE<br/>
*==SCons/Tool/MSCommon/Common.py==*<br/>
PATH [minimal extended]<sup>3</sup>
COMSPEC
OS
VS170COMNTOOLS
VS160COMNTOOLS
VS150COMNTOOLS
VS140COMNTOOLS
VS120COMNTOOLS
VS110COMNTOOLS
VS100COMNTOOLS
VS90COMNTOOLS
VS80COMNTOOLS
VS71COMNTOOLS
VSCOMNTOOLS
MSDevDir
VSCMD_DEBUG
VSCMD_SKIP_SENDTELEMETRY
windir
VCPKG_DISABLE_METRICS
VCPKG_ROOT<br>
*==Proposed SCons PR 4720==*<br/>
POWERSHELL_TELEMETRY_OPTOUT
PSDisableModuleAnalysisCacheCleanup
PSModuleAnalysisCachePath
PSModulePath [subset]<sup>4</sup>
]
B[
**Call MSVC Batch File**<br/>
vcvars64.bat & set
]
C[
**Extend Caller SCons env["ENV"]**<br>
*==SCons/Tool/MSCommon/Common.py==*<br/>
INCLUDE
LIB
LIBPATH
PATH
VSCMD_ARG_app_plat
VCINSTALLDIR
VCToolsInstallDir
]
A-->B;
B-->C;
Diagram footnotes:
Notes:
Edit [2025-05-29] to reflect PR code changes:
|
…PATH and add limited known shell PSModulePath paths. Changes: * Add the pwsh and powershell paths to the msvc environment PATH in the order discovered on the shell environment PATH. * Add the pwsh and powershell AllUsers and installation PSModule paths to the msvc environment PSModulePath in the order discovered on the shell environment PSModulePath.
# Manually resolved conflicts: # RELEASE.txt
Partially restore restore original GH #4717 CHANGES.txt text as a placeholder for reworked description.
Contributor Checklist:
CHANGES.txtandRELEASE.txt(and read theREADME.rst).