I wanted to use Nvidia visual profiler, which requires debugging info. to be generated for some performance analysis. So I passed nvcc debugging flags to cmake, using the following command line (-DCMAKE_BUILD_TYPE:STRING=Debug alone does not populate "-g -G" to nvcc somehow, which may be another bug):
..path/RAJAProxies.git/buildCuda] cmake -DENABLE_CUDA=on -DENABLE_OPENMP=On -DCMAKE_BUILD_TYPE:STRING=Debug -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCUDA_NVCC_FLAGS_DEBUG:STRING="-g -G" ../
However, nvcc will generate compilation errors if the -DCUDA_NVCC_FLAGS_DEBUG:STRING="-g -G" is used.
-- Generating /path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/CMakeFiles/lulesh-v2.0-RAJA-cuda.exe.dir//./lulesh-v2.0-RAJA-cuda.exe_generated_lulesh-cuda.cpp.o
/usr/tce/packages/cuda/cuda-9.2.148/bin/nvcc /path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp -x=cu -c -o /path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/CMakeFiles/lulesh-v2.0-RAJA-cuda.exe.dir//./lulesh-v2.0-RAJA-cuda.exe_generated_lulesh-cuda.cpp.o -m64 -DUSE_OPENMP -DUSE_CUDA -DUSE_CASE=9 -DUSE_MPI=0 -DLULESH_DEVICE=device -Xcompiler -fopenmp -restrict -arch sm_35 -std c++11 --expt-extended-lambda -ccbin /usr/tcetmp/bin/c++ -g -G -lineinfo -DNVCC -I/usr/tce/packages/cuda/cuda-9.2.148/include -I/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/lulesh-v2.0/RAJA -I/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/tpl/RAJA/include -I/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/tpl/RAJA/include -I/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/tpl/RAJA/tpl/cub
nvcc warning : '--device-debug (-G)' overrides '--generate-line-info (-lineinfo)'
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(288): error: calling a host function("Domain::p") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(288): error: identifier "Domain::p" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(288): error: calling a host function("Domain::q") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(288): error: identifier "Domain::q" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(516): error: calling a host function("Domain::nodelist") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(516): error: identifier "Domain::nodelist" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(528): error: calling a host function("CollectDomainNodesToElemNodes") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(528): error: identifier "CollectDomainNodesToElemNodes" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(560): error: calling a host function("Domain::nodeElemCount") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(560): error: identifier "Domain::nodeElemCount" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(561): error: calling a host function("Domain::nodeElemCornerList") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(561): error: identifier "Domain::nodeElemCornerList" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(571): error: calling a host function("Domain::fx") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(571): error: identifier "Domain::fx" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(572): error: calling a host function("Domain::fy") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(572): error: identifier "Domain::fy" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(573): error: calling a host function("Domain::fz") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(573): error: identifier "Domain::fz" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(758): error: calling a host function("Domain::nodelist") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(758): error: identifier "Domain::nodelist" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(819): error: calling a host function("Domain::ss") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(819): error: identifier "Domain::ss" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(820): error: calling a host function("Domain::elemMass") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(820): error: identifier "Domain::elemMass" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(832): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(832): error: identifier "Domain::xd" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(833): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(833): error: identifier "Domain::xd" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(834): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(834): error: identifier "Domain::xd" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(835): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(835): error: identifier "Domain::xd" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(836): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(836): error: identifier "Domain::xd" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(837): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
....
I wanted to use Nvidia visual profiler, which requires debugging info. to be generated for some performance analysis. So I passed nvcc debugging flags to cmake, using the following command line (-DCMAKE_BUILD_TYPE:STRING=Debug alone does not populate "-g -G" to nvcc somehow, which may be another bug):
..path/RAJAProxies.git/buildCuda] cmake -DENABLE_CUDA=on -DENABLE_OPENMP=On -DCMAKE_BUILD_TYPE:STRING=Debug -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCUDA_NVCC_FLAGS_DEBUG:STRING="-g -G" ../
However, nvcc will generate compilation errors if the -DCUDA_NVCC_FLAGS_DEBUG:STRING="-g -G" is used.
-- Generating /path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/CMakeFiles/lulesh-v2.0-RAJA-cuda.exe.dir//./lulesh-v2.0-RAJA-cuda.exe_generated_lulesh-cuda.cpp.o
/usr/tce/packages/cuda/cuda-9.2.148/bin/nvcc /path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp -x=cu -c -o /path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/CMakeFiles/lulesh-v2.0-RAJA-cuda.exe.dir//./lulesh-v2.0-RAJA-cuda.exe_generated_lulesh-cuda.cpp.o -m64 -DUSE_OPENMP -DUSE_CUDA -DUSE_CASE=9 -DUSE_MPI=0 -DLULESH_DEVICE=device -Xcompiler -fopenmp -restrict -arch sm_35 -std c++11 --expt-extended-lambda -ccbin /usr/tcetmp/bin/c++ -g -G -lineinfo -DNVCC -I/usr/tce/packages/cuda/cuda-9.2.148/include -I/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/lulesh-v2.0/RAJA -I/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/tpl/RAJA/include -I/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/tpl/RAJA/include -I/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/tpl/RAJA/tpl/cub
nvcc warning : '--device-debug (-G)' overrides '--generate-line-info (-lineinfo)'
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(288): error: calling a host function("Domain::p") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(288): error: identifier "Domain::p" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(288): error: calling a host function("Domain::q") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(288): error: identifier "Domain::q" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(516): error: calling a host function("Domain::nodelist") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(516): error: identifier "Domain::nodelist" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(528): error: calling a host function("CollectDomainNodesToElemNodes") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(528): error: identifier "CollectDomainNodesToElemNodes" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(560): error: calling a host function("Domain::nodeElemCount") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(560): error: identifier "Domain::nodeElemCount" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(561): error: calling a host function("Domain::nodeElemCornerList") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(561): error: identifier "Domain::nodeElemCornerList" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(571): error: calling a host function("Domain::fx") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(571): error: identifier "Domain::fx" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(572): error: calling a host function("Domain::fy") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(572): error: identifier "Domain::fy" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(573): error: calling a host function("Domain::fz") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(573): error: identifier "Domain::fz" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(758): error: calling a host function("Domain::nodelist") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(758): error: identifier "Domain::nodelist" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(819): error: calling a host function("Domain::ss") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(819): error: identifier "Domain::ss" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(820): error: calling a host function("Domain::elemMass") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(820): error: identifier "Domain::elemMass" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(832): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(832): error: identifier "Domain::xd" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(833): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(833): error: identifier "Domain::xd" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(834): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(834): error: identifier "Domain::xd" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(835): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(835): error: identifier "Domain::xd" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(836): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(836): error: identifier "Domain::xd" is undefined in device code
/path2file/username/workspace-wsa/tempdir/lulesh/RAJAProxies.git/buildCuda/lulesh-v2.0/RAJA/lulesh-cuda.cpp(837): error: calling a host function("Domain::xd") from a device function(" const") is not allowed
....