Spr based error estimation

[mortezah]

Summary

The changes in this PR are in regards to adaptive-solve implementation using spr-based error estimation. Additional updates/modifications are also included for things like solution transfer/field transfer/etc (see the detailed list at the end of this notes)

At this point, things work for both 2D/3D. There are a few new parameters introduced in the input file as follows:

1. ispradapt (default 0): 1 to turn on spr-adapt and 0 to turn it off
2. isprntime (default 10): an integer bigger than 0 to call spr-adapt every isprntime time steps
3. isprweight (default 0.1): a real number smaller than 1 specifying the weight of error to calculate the size-field (note smaller weight results in much more aggressive adaptation).
4. isprmaxsize (default 0.05): user-specified maximum allowed edge length in the entire mesh
5. isprrefinelevel (default 1): maximum level of refinement allowed during each call to spr-adapt [for example a value of 3 means the lengths of the edges will not reduce by a factor of more than 8(= 2^3) in each call to spr-adapt
6. isprcoarsenlevel (default -1): maximum level of coarsening allowed during each call to spr-adapt. use -1 to disable coarsening altogether [for example a value of 3 means the lengths of the edges will not increase by a factor of more than 8(= 2^3) in each call to spr-adapt

The high-level API that can be called on the Fortran side is adapt_by_spr. An example use is as follows

call adapt_by_spr(field_vec%id, psi_g, ntime, isprweight, isprmaxsize, isprrefinelevel, isprcoarsenlevel)

where field_vec contains the list of all the privacy variables (e.g. u_field, vz_field, chi_field, psi_field, etc) and the the second argument psi_g tells adapt_by_spr to use the psi_field for error estimation.

Checklist

• add API for setting verbosity level of pumi print statements (useful for debugging mesh adapt)
• fix the obvious memory leaks in solution-transfer code
• update create_field and create_vector subroutines to take an optional prefix argument as the field name
• add new APIs (both on the Fortran side and C++ side) to mark specific fields for solution transfer
• add a new API in output.f90 to mark the fields for solution transfer following the same logic used to write fields to restart
• add the basic spr routine to make sure you can run successive adapt solve loops [use any field for now]
• update the spr routine to use the "psi" field instead.
• add inputs for spr (weight, max_refine_level, max_coarsen_level, max_size)
• merge master to this branch and do the tests (on stellar)
• regression tests (on SCOREC)
• regression tests (on STELLAR) (regression tests pass on stellar except for the adapt that has small differences in values)
• restart works for cases 1,2,3, and 4
• restart does not work for case 5

• case 1: 2D restart after a no-adapt 2D run
• case 2: 2D restart after an adaptive 2D run
• case 3: 3D restart after an adaptive 2D run
• case 4: 3D restart after a no-adapt 3D run
• case 5: 3D restart after an adaptive 3D run [see notes below]

For case 5 the observations is as follows, If I let an adaptive run (with and adapt and output every 5 time step) run until time step 101, the lest time step of the solution (i.e., 101) completes successfully, and the last output file is for time step 100. If I restart another run from time step 100 then the solution at time step 101 does not converge.

NOTE when merging all the installations of m3dc1_socrec libraries have to updated otherwise users of the Fortran code will have trouble compiling it.

NOTE since some of the changes in this branch have been merged and reverted in the past (see commit 3ac2071) when merging to master might cause some issues. I have already merged the master branch (with that revert reverted) into this branch so hopefully merging this into master would not cause any issues.

NOTE Since I didn't have access to all the productions machines, I do not know if the regression tests pass. Since none of the regression tests relly on new spr-based adaptive procedures, I would guess that regression tests should pass. However I think to be safe someone with access to all the production machines needs to check that.

[mortezah]

@seegyoung @usmanriaz07 @bclyons12 @nferraro

This pull request includes all the changes I made regarding spr-base adapt. At the moment the only thing that does not work is restarting for case 5 [described in the body comment above].

Since I have started a new job I can only spend limited time over the next few days fixing that issue.