Load balancing, Surface-Sampling issues and Region support

[FoamScience]

This is in reference to #10

It seems blastAMR LB engine suffers a bug when remapping MeshObjects from old to new, load-balanced mesh. One effect is a segfault when using isoSurface-like surface sampling.

I wish the local surface was written by each MPI processor, but the OpenFOAM devs opted to reduce the information, then write the reconstructed surface. This writing phase is where an attempt is made to use an invalid MeshObject.

It seems any potential fix will be specific to this case (eg. fixing the MeshObjects by type). Fine but not optimal

References were made to similar utilities: multiDimAMR which we are of course aware of:

• blastAMR handles generic cell types, so the algorithms are slightly different from hex-based framework. Currently, we have an unusable "hex refinement engine" which we do plan to bring up to speed soon.
• Multi-region support is untested territory; but in principle there is no inherent reason for blastAMR to not run in multi-region context. The only requirement is that the solvers must support dynamic meshes. This is not the case for chtMultiRegionFoam which instantiates fvMeshes (instead of dynamicFvMesh objects). So alternative implementations must sought out (eg. chMultiRegionFoam from multiDimAMR
  • The alternative solver implementation must then reconstruct and correct fluxes properly for load-balancing context.
  • We probably can get away by protected (coupled) patches from refinement. blastAMR already has this functionality. But it's unclear how the multi-region solvers would handle lb-induced patch modifications; It's probably a solver-specific issue (or, a BC-related one).

[MontelukastNa]

It proves to be a little difficult to fix the segfaulting surface sampling.

For now, I recommend the following workaround (pseudo shell code):

for ti in $(foamListTimes -case processor0); do mpirun -np -parallel -postProcess -time "$ti"; done
Effects of this workaroud:

• Must disable the surfaces function object when running the solver
• Run the function object as a post-processing tool. It will only write surfaces for the output time directories

Originally posted by @FoamScience in #10

Dear M. Elwardi Fadeli

I’d like to report the isosurface sampling results based on the output time directories.
💡To begin with, the surfaces function object was deliberately disabled during the solver run to avoid runtime isosurface extraction, as previously discussed. Instead, isosurface sampling was performed afterwards using data from the output time directories.

💡So, I set up the sample dict system/surfaces as follows:

/*--------------------------------*- C++ -*----------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Version:  v2312
    \\  /    A nd           | Website:  www.openfoam.com
     \\/     M anipulation  |
-------------------------------------------------------------------------------
Description
    Writes out surface files with interpolated field data in VTK format, e.g.
    cutting planes, iso-surfaces and patch boundary surfaces.

    This file includes a selection of example surfaces, each of which the user
    should configure and/or remove.

\*---------------------------------------------------------------------------*/

type            surfaces;
libs            ("libsampling.so");

writeControl    writeTime;

surfaceFormat   vtp;
interpolationScheme cellPoint;

fields       (alpha.water);

isosurface
{
    type            isoSurfaceCell;
    interpolate     true;
}

surfaces
(
    p100
    {
        $isosurface;
        isoField    alpha.water;
        isoValue    0.5;
    }
);

💡Subsequently, the following commands were executed to perform isosurface extraction in parallel:

. $WM_PROJECT_DIR/bin/tools/RunFunctions
runParallel -o postProcess -func surfaces

❗However, this resulted in the following error message: (Since the message is too long, I’ve uploaded it as an attachment.)
log.postProcess.txt

Create time

Create mesh for time = 0

Sampled surface:
    p100 -> vtk
        isoSurface: p100 : isoMethod:cell regularise:full snap:1 field:alpha.water value:(0.5)

Time = 0

Reading fields:
    volScalarField: alpha.water

Executing functionObjects

Time = 0.001

Reading fields:
    volScalarField: alpha.water

Executing functionObjects
[10] [stack trace]
[10] =============
[13] [stack trace]
[13] =============
[15] [stack trace]
[15] =============
[18] [stack trace]
[18] =============
[19] [stack trace]
[19] =============
[5] [stack trace]
[5] =============
[7] [stack trace]
[7] =============
[12] [stack trace]
[12] =============
[1] [stack trace]
[1] =============
[4] [stack trace]
[4] =============
[9] [stack trace]
[9] =============
[2] [stack trace]
[2] =============
[10] #1  Foam::sigSegv::sigHandler(int)[13] #1  Foam::sigSegv::sigHandler(int)[15] #1  Foam::sigSegv::sigHandler(int)[7] #1  Foam::sigSegv::sigHandler(int)[19] #1  [5] #1  Foam::sigSegv::sigHandler(int)[18] #1  Foam::sigSegv::sigHandler(int)Foam::sigSegv::sigHandler(int)[12] #1  [1] #1  [4] #1  Foam::sigSegv::sigHandler(int)Foam::sigSegv::sigHandler(int)Foam::sigSegv::sigHandler(int)[9] #1  Foam::sigSegv::sigHandler(int)[2] #1  Foam::sigSegv::sigHandler(int) in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[7] #2  ? in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[15] # in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
2  ?[13] #2  ? in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[10] #2  ? in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[5] #2  ? in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[9] #2  ? in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[19] #2  ? in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[2] #2  ? in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[4] #2  ? in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[18] #2  ? in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[1] #2  ? in ~/OpenFOAM/OpenFOAMv2312/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[12] #2  ? in /lib/x86_64-linux-gnu/libpthread.so.0
 in /lib/x86_64-linux-gnu/libpthread.so.0
 in /lib/x86_64-linux-gnu/libpthread.so.0
 in /lib/x86_64-linux-gnu/libpthread.so.0

I rarely use the postProcess tool and therefore am not confident that my setup of extracting sampling surfaces from the output time directories is completely correct. Nevertheless, I have consulted a considerable amount of reference material regarding the use of postProcess, and I think that both the sampleDict setup and the command-line execution are reasonable.

That being said, the issue seems to persist regardless of whether the sampling is performed during runtime or applied retrospectively to the output data.

🥂This note summarizes my current test procedure and observations for your reference. Any guidance or suggestions would be greatly appreciated.

💡Additionally, during testing, I tried using both reconstructParMesh and reconstructPar because they enable manual output of smoother isosurfaces such as alpha.water.
While reconstructParMesh works fine, running reconstructPar produces the following error:

--> FOAM FATAL ERROR: 
Processor 0 has a different polyMesh at time 0.08 compared to any previous processors.
Please check time 0.08 directories on all processors for consistent mesh files.

This issue has also been reported in Henning’s multiDimAMR project [multiDimAMR-reconstructPar]HenningScheufler/multiDimAMR#5, where it occurred when enableBalancing was set to true, which is similar to the setting I used in blastAMR.

Looking forward to future progress, many thanks for your work on this!

Best regards,
Na