NVHPC runtime error with standalone MARBL run

[sjsprecious]

@mnlevy1981 identified a runtime error when using nvhpc compiler with -O2 flag. When switching to -O0 flag, NVHPC ran but the answer differed from the other compilers for conservation check.

Acceptance Criteria

• Make NVHPC work with -O2 and provide consistent results with Intel/GNU.
• Report to NVIDIA if it is a compiler bug and wait for their fix.

Ideas

• Debug more about the -O2 flag.

[mnlevy1981]

I've traced the location of the error to a where statement within an associate statement, and switching to an if statement inside a do loop solves the issue with -O2:

-        where (VNtot(auto_ind,:) > c0)
-          NO3_V(auto_ind,:) = (VNO3(auto_ind,:) / VNtot(auto_ind,:)) * photoC(auto_ind,:) * Q
-          NH4_V(auto_ind,:) = (VNH4(auto_ind,:) / VNtot(auto_ind,:)) * photoC(auto_ind,:) * Q
-        else where
-          NO3_V(auto_ind,:) = c0
-          NH4_V(auto_ind,:) = c0
-        end where
+        do k=1,size(VNtot,2)
+          if (VNtot(auto_ind,k) > c0) then
+            NO3_V(auto_ind,k) = (VNO3(auto_ind,k) / VNtot(auto_ind,k)) * photoC(auto_ind,k) * Q
+            NH4_V(auto_ind,k) = (VNH4(auto_ind,k) / VNtot(auto_ind,k)) * photoC(auto_ind,k) * Q
+          else
+            NO3_V(auto_ind,k) = c0
+            NH4_V(auto_ind,k) = c0
+          end if

-        where (VPtot(auto_ind,:) > c0)
-          PO4_V(auto_ind,:) = (VPO4(auto_ind,:) / VPtot(auto_ind,:)) * photoC(auto_ind,:) * gQp(auto_ind,:)
-          DOP_V(auto_ind,:) = (VDOP(auto_ind,:) / VPtot(auto_ind,:)) * photoC(auto_ind,:) * gQp(auto_ind,:)
-        else where
-          PO4_V(auto_ind,:) = c0
-          DOP_V(auto_ind,:) = c0
-        end where
+          if (VPtot(auto_ind,k) > c0) then
+            PO4_V(auto_ind,k) = (VPO4(auto_ind,k) / VPtot(auto_ind,k)) * photoC(auto_ind,k) * gQp(auto_ind,k)
+            DOP_V(auto_ind,k) = (VDOP(auto_ind,k) / VPtot(auto_ind,k)) * photoC(auto_ind,k) * gQp(auto_ind,k)
+          else
+            PO4_V(auto_ind,k) = c0
+            DOP_V(auto_ind,k) = c0
+          end if
+        end do

unfortunately, compiling with -O2 now gives the same conservation check error I was getting with the -O0 workaround

[mnlevy1981]

I've found at least one issue with how nvfortran compiles associate statements, which we use throughout MARBL. Specifically, there is an issue with using associate() to create a shortcut to a sub-array when the sub-array uses a variable index in one of the dimensions of the full array:

program associate_statements

  real, dimension(3,4) :: mat
  integer :: i, j

  do j=1,4 ; do i=1,3
    mat(i,j) = 10*i + j
  end do ; end do

  i=2
  associate(arr => mat(i,:))
    print*, mat(i,:)
    print*, ""
    print*, arr
  end associate

end program associate_statements

GNU (12.4) output:

$ gfortran associate_statements.F90
$ ./a.out
   21.0000000       22.0000000       23.0000000       24.0000000

   21.0000000       22.0000000       23.0000000       24.0000000

Intel (2024.2.1) output:

$ ifx associate_statements.F90
$ ./a.out
   21.00000       22.00000       23.00000       24.00000

   21.00000       22.00000       23.00000       24.00000

NVHPC (25.7) output:

$ nvfortran associate_statements.F90
$ ./a.out
    21.00000        22.00000        23.00000        24.00000

    0.000000        11.00000        21.00000        31.00000

Note that nvfortran provides the correct output with

  associate(arr => mat(2,:))

The problem crops up when trying to associate with the index i.

---

We use the construct throughout the MARBL code, and I don't think it is worth trying to re-write the library to avoid it... once there is a compiler version that treats this correctly I can continue testing.

[mnlevy1981]

@sjsprecious are you in touch with someone in the compiler group at NVIDIA? Do you know the process for flagging this issue so it can get fixed? Note that there is a very simple (17 line) example of the bug in #19 (comment)

[sjsprecious]

Thanks @mnlevy1981 for your update. This is great progress!

We can forward your reproducer to our NVIDIA collaborators and hopefully they can fix it in their next release.

[mnlevy1981]

I'm going to close this issue; I am hopeful that getting this nvfortran bug fixed will let us run MARBL w/ the NVIDIA compiler, but if it doesn't we can either re-open this issue or create a new one specific to whatever else is going wrong.