Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Simplify the cpack options in AthenaUtils.archiveWithCpack #26

Open
krasznaa opened this issue Apr 24, 2023 · 11 comments
Open

Simplify the cpack options in AthenaUtils.archiveWithCpack #26

krasznaa opened this issue Apr 24, 2023 · 11 comments

Comments

@krasznaa
Copy link

Since there was only a limited reaction on the corresponding ATLAS e-mail thread, I thought it would be better to open an issue here.

The following way of calling CPack is way "over engineered". 😦

https://github.com/PanDAWMS/panda-client/blob/master/pandaclient/AthenaUtils.py#L1067-L1070

Most of the -D arguments are unnecessary in there. But there is one that is actively disruptive: -D CPACK_INSTALL_CMAKE_PROJECTS="" That one explicitly tells CPack not to install anything.

With the way that AtlasCMake has operated for years, this option was simply ignored. But now that I'm trying to make the ATLAS builds behave in a "more standard" way, this flag prevents prun from collecting the code of the users that they are trying to execute a job with.

Please remove that argument from the CPack call. It was not needed with old ATLAS analysis releases either, and is now actively hurtful with the latest ones.
@tmaeno
Copy link
Contributor

tmaeno commented Apr 24, 2023

Removed in 1.5.45. Will be deployed on CVMFS shortly.

@tmaeno tmaeno closed this as completed Apr 24, 2023
@tmaeno
Copy link
Contributor

tmaeno commented Apr 24, 2023 via email

@tmaeno
Copy link
Contributor

tmaeno commented Apr 24, 2023 via email

@tmaeno
Copy link
Contributor

tmaeno commented Apr 24, 2023

With 22.0.50,Athena

$ cpack -B /afs/cern.ch/user/t/tmaeno/Offline/current/run/72a0f5af-245c-418b-a646-ddd44d057a88 -D CPACK_PACKAGE_FILE_NAME=sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70 -G TGZ -D CPACK_PACKAGE_NAME="" -D CPACK_PACKAGE_VERSION="" -D CPACK_PACKAGE_VERSION_MAJOR="" -D CPACK_PACKAGE_VERSION_MINOR="" -D CPACK_PACKAGE_VERSION_PATCH="" -D CPACK_PACKAGE_DESCRIPTION=""
CPack Error: Please specify build tree of the project that uses CMake using CPACK_INSTALL_CMAKE_PROJECTS, specify CPACK_INSTALL_COMMANDS, CPACK_INSTALL_SCRIPT, or CPACK_INSTALLED_DIRECTORIES.

while

$ cpack -B /afs/cern.ch/user/t/tmaeno/Offline/current/run/72a0f5af-245c-418b-a646-ddd44d057a88 -D CPACK_PACKAGE_FILE_NAME=sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70 -G TGZ -D CPACK_PACKAGE_NAME="" -D CPACK_PACKAGE_VERSION="" -D CPACK_PACKAGE_VERSION_MAJOR="" -D CPACK_PACKAGE_VERSION_MINOR="" -D CPACK_PACKAGE_VERSION_PATCH="" -D CPACK_PACKAGE_DESCRIPTION="" -D CPACK_INSTALL_CMAKE_PROJECTS=""
CPack: Create package using TGZ
CPack: Install projects
CPack: Create package
CPack: - package: /afs/cern.ch/user/t/tmaeno/Offline/current/run/72a0f5af-245c-418b-a646-ddd44d057a88/sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70.tar.gz generated.

Apparently -D CPACK_INSTALL_CMAKE_PROJECTS="" is required for old releases.

@krasznaa , what should we do then?

@tmaeno tmaeno reopened this Apr 24, 2023
@krasznaa
Copy link
Author

Curious. Is prun not executing the cpack command inside of the directory that it deemed as the "build directory"? Because it really should. That would make its own work a whole lot easier.

I.e. when executing the type of command that you posted Tadashi inside of my "build directory", I get:

Singularity> cpack -B /tmp/krasznaa/ -D CPACK_PACKAGE_FILE_NAME=sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70 -G TGZ -D CPACK_PACKAGE_NAME="" -D CPACK_PACKAGE_VERSION="" -D CPACK_PACKAGE_VERSION_MAJOR="" -D CPACK_PACKAGE_VERSION_MINOR="" -D CPACK_PACKAGE_VERSION_PATCH="" -D CPACK_PACKAGE_DESCRIPTION=""
CPack: Create package using TGZ
CPack: Install projects
CPack: - Install scripts: /cvmfs/atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallArea/x86_64-centos7-gcc11-opt/cmake/modules/scripts/cpack_install.cmake
CPack: - Install script: /cvmfs/atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallArea/x86_64-centos7-gcc11-opt/cmake/modules/scripts/cpack_install.cmake
CPack: Create package
CPack: - package: /tmp/krasznaa/sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70.tar.gz generated.
Singularity>

But if I try to execute it from a different directory, sure enough, I get:

Singularity> cd ..
Singularity> cpack -B /tmp/krasznaa/ -D CPACK_PACKAGE_FILE_NAME=sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70 -G TGZ -D CPACK_PACKAGE_NAME="" -D CPACK_PACKAGE_VERSION="" -D CPACK_PACKAGE_VERSION_MAJOR="" -D CPACK_PACKAGE_VERSION_MINOR="" -D CPACK_PACKAGE_VERSION_PATCH="" -D CPACK_PACKAGE_DESCRIPTION=""
CPack Error: Please specify build tree of the project that uses CMake using CPACK_INSTALL_CMAKE_PROJECTS, specify CPACK_INSTALL_COMMANDS, CPACK_INSTALL_SCRIPT, or CPACK_INSTALLED_DIRECTORIES.
Singularity>

But then again, when being outside of the "build directory", and specifying a -B argument to the command to tell it to use some other directory as the target of the TGZ file, I could not make the command work at all. 😕

So... where does Panda run these commands from exactly...?

@krasznaa
Copy link
Author

Also, this line in Asoka's log is pretty worrisome:

INFO : The build directory is /cvmfs/atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallArea

The code should really be able to recognize when no build directory exists. As in that case it just does not need to package up anything with CPack...

@tmaeno
Copy link
Contributor

tmaeno commented Apr 24, 2023 •
edited
Loading

I was asked to execute cpack from the first entry of $CMAKE_PREFIX_PATH.

INFO : The build directory is xyz

this message means cpack is executed from xyz. I suppose that the first entry of $CMAKE_PREFIX_PATH points to
user's build dir if the user has it and sets up the environment properly.
In Asoka's test, I guess he didn't have the build dir, cpack was executed from/cvmfs/atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallArea, and it used to create an empty tarball,
but now it fails.

If the easiest solution is to skip cpack when the user doesn't have own build dir, how can I check the presence of user's
build dir using env variables or something?

@krasznaa
Copy link
Author

To be correct, you execute cpack from ${CMAKE_PREFIX_PATH[0]}/... I.e. "one directory up" from the first entry in CMAKE_PREFIX_PATH. (This is what the extra os.path.dirname(...) does in: https://github.com/PanDAWMS/panda-client/blob/master/pandaclient/AthenaUtils.py#L1063)

This is correct. When the user actually uses a "build directory", that is indeed where the directory is. But when there's no "build directory" used, just an existing ATLAS release, that directory is usually some path on CVMFS.

I believe the best thing to do is to check whether there is a file called CPackConfig.cmake in the "prospective build directory". If there is such a file there, then that does look like a "build directory". But if not, then cpack will not know what to do anyway. As by default, unless told otherwise (which your command doesn't do), it looks for a file with that name. (In the current directory, or in the directory specified with -B.)

So if I understand correctly, the issue here is really just to not try to use cpack when it shouldn't be used. I still believe that all old releases will work with the modified cpack call as long as that call is made in an actual "build directory". 😉

@tmaeno
Copy link
Contributor

tmaeno commented Apr 25, 2023

OK, I've added the change in 1.5.46.

@tmaeno
Copy link
Contributor

tmaeno commented Apr 25, 2023 via email

@tmaeno
Copy link
Contributor

tmaeno commented Apr 25, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tmaeno @krasznaa