On ARM with flang

[LyndStringer]

On the Sandia NL ARM machine "stria", I tried to get flang working with the c_sw kernel. The command:

export CMAKE_PREFIX_PATH=/opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5

tells CMAKE where the netcdf directory is located. CMAKE then tries to tell flang where the netcdf directory is with the
"-isystem" flag. But flang ignores it, which causes an error in finding file: netcdf.mod. Is there flang on Hera or Orion?
If so, this would be easier to debug.

module swap gnu7 arm/20.0
source /projects/marvell/setup_env_llvm

rm -rf build ; mkdir build ; cd build
export CC=clang
export FC=flang
export CMAKE_PREFIX_PATH=/opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5

[lmstrin@stria-login2 build]$ cmake -DCMAKE_BUILD_TYPE=release ..
-- The Fortran compiler identification is Flang 99.99.1
-- The C compiler identification is Clang 9.0.1
-- Check for working Fortran compiler: /projects/marvell/llvm/9.0.1/bin/flang
-- Check for working Fortran compiler: /projects/marvell/llvm/9.0.1/bin/flang -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /projects/marvell/llvm/9.0.1/bin/flang supports Fortran 90
-- Checking whether /projects/marvell/llvm/9.0.1/bin/flang supports Fortran 90 -- yes
-- Check for working C compiler: /projects/marvell/llvm/9.0.1/bin/clang
-- Check for working C compiler: /projects/marvell/llvm/9.0.1/bin/clang -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found OpenMP_C: -fopenmp=libomp (found version "3.1")
-- Could NOT find OpenMP_Fortran (missing: OpenMP_Fortran_FLAGS OpenMP_Fortran_LIB_NAMES)
-- Could NOT find OpenMP (missing: OpenMP_Fortran_FOUND Fortran) (found version "3.1")
NetCDF_C_INCLUDE_FILE: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/include/netcdf.h
NetCDF_Fortran_INCLUDE_FILE: /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/include/netcdf.mod
NetCDF_C_CONFIG_EXECUTABLE: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/bin/nc-config
NetCDF_Fortran_CONFIG_EXECUTABLE: /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/bin/nf-config
NetCDF_C_LIBRARY: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/lib/libnetcdf.so
NetCDF_Fortran_LIBRARY: /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/lib/libnetcdff.so
-- Found NetCDF: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/include;/opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/include (found version "4.6.3") found components: C Fortran
-- FindNetCDF defines targets:
-- - NetCDF_VERSION [4.6.3]
-- - NetCDF_PARALLEL [TRUE]
-- - NetCDF_C_CONFIG_EXECUTABLE [/opt/atse/libs/arm/openmpi4/netcdf/4.6.3/bin/nc-config]
-- - NetCDF::NetCDF_C [SHARED] [Root: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3] Lib: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/lib/libnetcdf.so
-- - NetCDF_Fortran_CONFIG_EXECUTABLE [/opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/bin/nf-config]
-- - NetCDF::NetCDF_Fortran [SHARED] [Root: /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5] Lib: /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/lib/libnetcdff.so
-- Configuring done
-- Generating done
-- Build files have been written to: /ascldap/users/lmstrin/SENA-c_sw/ref/build
[lmstrin@stria-login2 build]$ make VERBOSE=1
/opt/atse/utils/cmake/3.12.2/bin/cmake -H/ascldap/users/lmstrin/SENA-c_sw/ref -B/ascldap/users/lmstrin/SENA-c_sw/ref/build --check-build-system CMakeFiles/Makefile.cmake 0
/opt/atse/utils/cmake/3.12.2/bin/cmake -E cmake_progress_start /ascldap/users/lmstrin/SENA-c_sw/ref/build/CMakeFiles /ascldap/users/lmstrin/SENA-c_sw/ref/build/CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all
make[1]: Entering directory /home/lmstrin/SENA-c_sw/ref/build' make -f src/CMakeFiles/c_sw.dir/build.make src/CMakeFiles/c_sw.dir/depend make[2]: Entering directory /home/lmstrin/SENA-c_sw/ref/build'
cd /ascldap/users/lmstrin/SENA-c_sw/ref/build && /opt/atse/utils/cmake/3.12.2/bin/cmake -E cmake_depends "Unix Makefiles" /ascldap/users/lmstrin/SENA-c_sw/ref /ascldap/users/lmstrin/SENA-c_sw/ref/src /ascldap/users/lmstrin/SENA-c_sw/ref/build /ascldap/users/lmstrin/SENA-c_sw/ref/build/src /ascldap/users/lmstrin/SENA-c_sw/ref/build/src/CMakeFiles/c_sw.dir/DependInfo.cmake --color=
Dependee "/ascldap/users/lmstrin/SENA-c_sw/ref/build/src/CMakeFiles/c_sw.dir/DependInfo.cmake" is newer than depender "/ascldap/users/lmstrin/SENA-c_sw/ref/build/src/CMakeFiles/c_sw.dir/depend.internal".
Dependee "/ascldap/users/lmstrin/SENA-c_sw/ref/build/src/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/ascldap/users/lmstrin/SENA-c_sw/ref/build/src/CMakeFiles/c_sw.dir/depend.internal".
Scanning dependencies of target c_sw
make[2]: Leaving directory /home/lmstrin/SENA-c_sw/ref/build' make -f src/CMakeFiles/c_sw.dir/build.make src/CMakeFiles/c_sw.dir/build make[2]: Entering directory /home/lmstrin/SENA-c_sw/ref/build'
[ 16%] Building Fortran object src/CMakeFiles/c_sw.dir/netCDFModule.f90.o
cd /ascldap/users/lmstrin/SENA-c_sw/ref/build/src && /projects/marvell/llvm/9.0.1/bin/flang -DNDEBUG -isystem /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/include -isystem /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/include -O3 -DNDEBUG -c /ascldap/users/lmstrin/SENA-c_sw/ref/src/netCDFModule.f90 -o CMakeFiles/c_sw.dir/netCDFModule.f90.o
clang-9: warning: argument unused during compilation: '-isystem /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/include' [-Wunused-command-line-argument]
clang-9: warning: argument unused during compilation: '-isystem /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/include' [-Wunused-command-line-argument]
F90-F-0004-Unable to open MODULE file netcdf.mod (/ascldap/users/lmstrin/SENA-c_sw/ref/src/netCDFModule.f90: 7)
F90/aarch64 Linux Flang - 1.5 2017-05-01: compilation aborted
make[2]: *** [src/CMakeFiles/c_sw.dir/netCDFModule.f90.o] Error 1
make[2]: Leaving directory /home/lmstrin/SENA-c_sw/ref/build' make[1]: *** [src/CMakeFiles/c_sw.dir/all] Error 2 make[1]: Leaving directory /home/lmstrin/SENA-c_sw/ref/build'
make: *** [all] Error 2
[lmstrin@stria-login2 build]$ ls /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/include
netcdf_aux.h netcdf_filter.h netcdf.h netcdf_mem.h netcdf_meta.h netcdf_par.h

[christopherwharrop-noaa]

@LyndStringer - Please edit this issue to include a description of the problem and how to reproduce it.

[LyndStringer]

At the bottom of the long comment, I "ls"ed the wrong directory. Here is the directory that contains netcdf.mod. This was also denoted in an -isystem flag, but ignored, so netcdf.mod was not found.

[lmstrin@stria-login2 build]$ ls /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/include
netcdf_aux.h netcdf_filter.h netcdf.h netcdf_mem.h netcdf_meta.h netcdf_par.h

[LyndStringer]

While getting clang/flang working with out kernels is a desirable goal, I doubt we'll be able to get flang/clang working with the
kernels by Thursday. There are a number of issues beyond cmake's -isystem problem on stria. So in my mind, this is not urgent.

[christopherwharrop-noaa]

@LyndStringer - You still ls'ed the wrong directory. Does /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/include contain the netcdf.mod file?

[christopherwharrop-noaa]

@LyndStringer - Also, clang/flang compilers are not currently supported. We don't even have options for that in the build flags

[christopherwharrop-noaa]

This appears to be a bug in flang and/or cmake. See: here.

You could try a newer version of cmake. But, probably, we'll have to find a way to force it to use -I instead of -system.

[christopherwharrop-noaa]

@LyndStringer - Try adding a custom setting for the Fortran flags to add the include path with -I:

cmake -DCMAKE_Fortran_FLAGS=-I/opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/include