Update LLVM/Clang to LLVM 16

[larryk85]

Currently, CDT is using LLVM 9 for its internal toolchain. This is a modified version of LLVM, so some care needs to be made during the upgrade process to ensure that all parts that are specific to CDT are carried over and up-ported if needed.

[EDIT]: Question the need to use a modified version of LLVM. Using a vanilla version would be simpler approach. Projects like Blanc have show this approach can be successfully. We lack the experts in compilers and the resourcing to make custom compiler workable. - EP

Since the LLVM project has changed how they 'house' their project on Github, i.e. no longer distinct repositories and now use a mono-repo, this will make things a bit more difficult. The easiest way to move forward I believe is to generate a Git patch from our LLVM code against the mono-repos directories with the mono-repo being checked out as llvmorg-9.0.0 branch. From there you will be able to apply the patch to the 9.0.0 branch (hopefully no issues exist here), then merge llvmorg-16.0.0 and address any issues.

Instead of forking llvm the same as we have in the past and creating cdt- repos, we will be simply adding it to the project directly.

[jolly-fellow]

I have compared llvmorg-9.0.0 tag from LLVM repo with main cdt-llvm repo and found these differences:

--- a/llvm/utils/release/build_llvm_package.bat
+++ b/utils/release/build_llvm_package.bat
@@ -26,8 +26,8 @@ set python64_dir=C:\Users%USERNAME%\AppData\Local\Programs\Python\Python36

set revision=%1
set branch=trunk
-set package_version=9.0.0-r%revision%
-set clang_format_vs_version=9.0.0.%revision%
+set package_version=9.0.1-r%revision%^M
+set clang_format_vs_version=9.0.1.%revision%^M
set build_dir=llvm_package_%revision%

--- a/llvm/utils/lit/lit/init.py
+++ b/utils/lit/lit/init.py
@@ -2,7 +2,7 @@

author = 'Daniel Dunbar'
email = 'daniel@minormatter.com'
-versioninfo = (0, 9, 0)
+versioninfo = (0, 9, 1)
version = '.'.join(str(v) for v in versioninfo) + 'dev'

all = []

--- a/llvm/utils/gn/secondary/llvm/version.gni
+++ b/utils/gn/secondary/llvm/version.gni
@@ -1,4 +1,4 @@
llvm_version_major = 9
llvm_version_minor = 0
-llvm_version_patch = 0
+llvm_version_patch = 1

Was it a try to switch to the next version?

[jolly-fellow]

The easiest way to move forward I believe is to generate a Git patch from our LLVM code against the mono-repos directories with the mono-repo being checked out as llvmorg-9.0.0 branch. From there you will be able to apply the patch to the 9.0.0 branch (hopefully no issues exist here), then merge llvmorg-16.0.0 and address any issues.

cdt-llvm repository main branch is an LLVM repository tag llvmorg-9.0.0 with changes from our team so I don't understand why not just merge llvmorg-16.0.0 into cdt-llvm main and then move cdt-llvm repo to cdt repo as a directory if it should be there.
P.S. But I'd prefer to leave cdt-llvm repository separated because clang and lld placed in separated repositories too. Or lets use and patch whole LLVM repository where versions of LLVM, clang and lld are synchronized. Separated repositories give flexibility to work with different versions of LLVM, clang and lld if you need it as now we work with new clang but old LLVM. I just not sure if you really need it.

[jolly-fellow]

We agreed with @larryk85 a procedure of the update in the Telegram:

@larryk85 -----------------------------------------------------------------------------------------------------
The separate repos is the way that LLVM used to have their projects. They changed to a mono repo around LLVM 10 I believe.

Yes that is the ask, to do away with the separated forked repos and just merge our changes onto the monorepo of LLVM some that we then just have one repo.

Also instead of forking the llvm repo and adding it as a submodule. What I would like to see is just adding the merged LLVM code directly to cdt. I.e. hoist and rebase into a new folder that we will just call llvm and do away with the cdt-llvm stuff.

That way we would be able to work on those parts and only have one PR. We would also be able to apply upstream changes to llvm-project directly to cdt.

@jolly-fellow -----------------------------------------------------------------------------------------------------
Where do you want to have llvm directory? The path.

@larryk85 -----------------------------------------------------------------------------------------------------
The same location as cdt-llvm. We'll just replace that with this one and we'd have to fix up some Cmake leading to it. I believe we rely on an old way of how LLVM used to build.

So if you do it and find out that it is only building llvm and not clang and lld. Then Google cmake for LLVM building and you'll see a section on the CMake variables to toggle on which subprojects to build.

@jolly-fellow -----------------------------------------------------------------------------------------------------
I'd prefer to put whole the code of LLVM monorepo to CTD. It avoids any problems with difference in building or merging because of difference in directory structure. Now we have clang and lld submodules included into cdt-llvm directory. In the monorepo they are on the same level as llvm itself and other utilities. IMHO it is better to synchronize the directory structure one time and then follow llvm-project structure instead to keep structure of cdt-llvm and have problems each time when we will need to update versions of LLVM.

git easily merge corresponding directories from lvm-project into our repositories, as I understand because these directories are roots of the corresponding repos. So only LLVM code will be merged from lvm-project into cdt-llvm repo, cmake code will be merged into cmake and lld to our lld.
So I propose to update the code into our repos step by step: llvm, then clang, then lld to the code from llvmorg-16.0.0 tag of the monorepo. We can merge and test each step separately. Then when whole the code will be updated we copy lvm-project into cdt repo, copy code from llvm, clang and lld repos to the corresponding dirs of lvm-project into cdt repo and update building system to the new dirs structure.
I believe each step will be small and controlled.
@larryk85 -----------------------------------------------------------------------------------------------------
We are saying the same thing 😊 yes continue with that.