SurfaceSampling and blastAMR

[JanGaertner]

There seems to be an error when you use surface sampling together with blastAMR.

When surface sampling is deactivated or if sampling occurs AFTER the first rebalance it works.

When surface sampling is enabled following error occurs:

PIMPLE: iteration 1
---->! custom error estimator !<----
---->! end    error estimator !<----
Protecting 2706 cells next to requested boundary patches.
Selected 1625 cells for refinement out of 236952.
Refined from 236952 to 241827 cells.
canBalance(): Maximum imbalance = 122.492939 %
Selecting decompositionMethod scotch [64]
Selecting decompositionConstraint polyRefinement
polyRefinement : setting constraints to preserve refinement history
polyRefinement : reading fvMeshPolyRefiner from time "3.87719e-06"
2D case detected. Adding prismatic2DRefinement topology modifier
polyRefinement : reading fvMeshPolyRefiner from time "3.87719e-06"
2D case detected. Adding prismatic2DRefinement topology modifier
1
Maximum imbalance = 122.492939 %
polyRefinement : reading fvMeshPolyRefiner from time "3.87719e-06"
2D case detected. Adding prismatic2DRefinement topology modifier
polyRefinement : reading fvMeshPolyRefiner from time "3.87719e-06"
2D case detected. Adding prismatic2DRefinement topology modifier
Mapping the fields ...
Distributing the mesh ...
Successfully distributed mesh
New max imbalance: 0.967220368%
[47] [61] 
[61] 
[61] --> FOAM FATAL ERROR: (openfoam-2012)
[61] Cannot dereference nullptr at index 10 in range [0,10)
[61] 
[61] 
[61]     From const T& Foam::UPtrList<T>::operator[](Foam::label) const [with T = Foam::polyPatch; Foam::label = long int]
[61]     in file /home/itvjg/OpenFOAM/OpenFOAM-v2012-int64/src/OpenFOAM/lnInclude/UPtrListI.H at line 218.
[61] 
FOAM parallel run aborting
[61] 
[61] #0  Foam::error::printStack(Foam::Ostream&)[33] #0  [35] #0  [36] #0  [37] #0  [46] #0  Foam::error::printStack(Foam::Ostream&)[59] #0  Foam::error::printStack(Foam::Ostream&)[60] #0  Foam::error::printStack(Foam::Ostream&)[62] #0  Foam::error::printStack(Foam::Ostream&)[63] #0  Foam::error::printStack(Foam::Ostream&)[32] #0  Foam::error::printStack(Foam::Ostream&)[12] #0  Foam::error::printStack(Foam::Ostream&)[14] #0  Foam::error::printStack(Foam::Ostream&)[17] #0  Foam::error::printStack(Foam::Ostream&)[18] #0  Foam::error::printStack(Foam::Ostream&)[19] #0  Foam::error::printStack(Foam::Ostream&)[20] #0  Foam::error::printStack(Foam::Ostream&)[21] #0  Foam::error::printStack(Foam::Ostream&)[24] #0  Foam::error::printStack(Foam::Ostream&)[25] #0  Foam::error::printStack(Foam::Ostream&)[26] #0  Foam::error::printStack(Foam::Ostream&)[31] #0  Foam::error::printStack(Foam::Ostream&)Foam::error::printStack(Foam::Ostream&)Foam::error::printStack(Foam::Ostream&)Foam::error::printStack(Foam::Ostream&)Foam::error::printStack(Foam::Ostream&)[53] #0  Foam::error::printStack(Foam::Ostream&)[0] #0  Foam::error::printStack(Foam::Ostream&)[4] #0  Foam::error::printStack(Foam::Ostream&)[5] #0  Foam::error::printStack(Foam::Ostream&)[8] #0  Foam::error::printStack(Foam::Ostream&)[9] #0  Foam::error::printStack(Foam::Ostream&)
[47] 
[47] --> FOAM FATAL ERROR: (openfoam-2012)
[47] Cannot dereference nullptr at index 10 in range [0,10)
[47] 
[47] 
[47]     From const T& Foam::UPtrList<T>::operator[](Foam::label) const [with T = Foam::polyPatch; Foam::label = long int]
[47]     in file /home/itvjg/OpenFOAM/OpenFOAM-v2012-int64/src/OpenFOAM/lnInclude/UPtrListI.H at line 218.
[47] 
FOAM parallel run aborting
[47] 
[47] #0  Foam::error::printStack(Foam::Ostream&) at ??:?
[46] #1  Foam::sigSegv::sigHandler(int) at ??:?
 at ??:?
[60] #1  Foam::sigSegv::sigHandler(int) at ??:?
[62] #1  Foam::sigSegv::sigHandler(int) at ??:?
[63] #1  Foam::sigSegv::sigHandler(int) at ??:?
[61] #1  Foam::error::exitOrAbort(int, bool) at ??:?
[36] #1  Foam::sigSegv::sigHandler(int) at ??:?
[37] #1  Foam::sigSegv::sigHandler(int) at ??:?
 at ??:?
 at ??:?
[59] #1  Foam::sigSegv::sigHandler(int) at ??:?
[53] #1  Foam::sigSegv::sigHandler(int)[35] #1  Foam::sigSegv::sigHandler(int)[32] #1  Foam::sigSegv::sigHandler(int)[33] #1  Foam::sigSegv::sigHandler(int) at ??:?
[9] #1  Foam::sigSegv::sigHandler(int) at ??:?
[8] #1  Foam::sigSegv::sigHandler(int) at ??:?
[24] #1  Foam::sigSegv::sigHandler(int) at ??:?
[25] #1  Foam::sigSegv::sigHandler(int) at ??:?
[26] #1  Foam::sigSegv::sigHandler(int) at ??:?
[4] #1  Foam::sigSegv::sigHandler(int) at ??:?
[12] #1  Foam::sigSegv::sigHandler(int) at ??:?
[14] #1  Foam::sigSegv::sigHandler(int) at ??:?
 at ??:?
 at ??:?
 at ??:?
[20] #1  Foam::sigSegv::sigHandler(int) at ??:?
[21] #1  Foam::sigSegv::sigHandler(int) at ??:?
 at ??:?
[5] #1   at ??:?
[17] #1  Foam::sigSegv::sigHandler(int)[18] #1  Foam::sigSegv::sigHandler(int)[19] #1  Foam::sigSegv::sigHandler(int)Foam::sigSegv::sigHandler(int) at ??:?
[0] #1  Foam::sigSegv::sigHandler(int)[31] #1  Foam::sigSegv::sigHandler(int)[47] #1  Foam::error::exitOrAbort(int, bool) at ??:?
[36] #2  ? at ??:?
[60] #2  ? at ??:?
[37] #2  ? at ??:?
[63] #2  ? at ??:?
[61] #2  void Foam::fvMeshBalance::correctBoundaries<Foam::GeometricField<double, Foam::pointPatchField, Foam::pointMesh> >() at ??:?
 at ??:?
[35] #2  ? at ??:?
[32] #2  ? at ??:?
[33] #2  ?[62] #2  ? at ??:?
[46] #2  ? at ??:?
[53] #2  ? at ??:?
[59] #2  ? at ??:?
[26] #2  ? at ??:?
[8] #2  ? at ??:?
[24] #2  ? at ??:?
[25] #2  ? at ??:?
[12] #2  ? at ??:?
[14] #2  ? at ??:?
[20] #2  ? at ??:?
[21] #2  ? at ??:?
[4] #2  ? in /lib/x86_64-linux-gnu/libc.so.6
[37] #3  __dynamic_cast at ??:?
 in /lib/x86_64-linux-gnu/libc.so.6
 at ??:?
[18] #2  ? at ??:?
[17] #2  ? at ??:?
[19] #2  ?[5] #2  ? at ??:?
[63] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[60] #3  __dynamic_cast[47] #2  void Foam::fvMeshBalance::correctBoundaries<Foam::GeometricField<double, Foam::pointPatchField, Foam::pointMesh> >() in /lib/x86_64-linux-gnu/libc.so.6
[62] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[32] #3  __dynamic_cast at ??:?
[61] #3  Foam::fvMeshBalance::distribute() in /lib/x86_64-linux-gnu/libc.so.6
[33] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[35] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[46] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[36] #3  __dynamic_cast at ??:?
[9] #2  ? at ??:?
[0] #2  ? at ??:?
[31] #2  ? in /lib/x86_64-linux-gnu/libc.so.6
[53] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[59] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[25] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[26] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[21] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[12] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[14] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[20] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
 in /lib/x86_64-linux-gnu/libc.so.6
[8] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[4] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
[17] #[18] #3  __dynamic_cast3  __dynamic_cast in /lib/x86_64-linux-gnu/libstdc++.so.6
[37] #4  void Foam::fvMeshBalance::correctBoundaries<Foam::GeometricField<double, Foam::pointPatchField, Foam::pointMesh> >() in /lib/x86_64-linux-gnu/libstdc++.so.6
 in /lib/x86_64-linux-gnu/libstdc++.so.6
[63] #4  void Foam::fvMeshBalance::correctBoundaries<Foam::GeometricField<double, Foam::pointPatchField, Foam::pointMesh> >()[60] #4  void Foam::fvMeshBalance::correctBoundaries<Foam::GeometricField<double, Foam::pointPatchField, Foam::pointMesh> >() in /lib/x86_64-linux-gnu/libc.so.6
[24] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libc.so.6
 in /lib/x86_64-linux-gnu/libc.so.6
[19] #3  __dynamic_cast[5] #3  __dynamic_cast in /lib/x86_64-linux-gnu/libstdc++.so.6
[62] #4  void Foam::fvMeshBalance::correctBoundaries<Foam::GeometricField<double, Foam::pointPatchField, Foam::pointMesh> >() in /lib/x86_64-linux-gnu/libstdc++.so.6
[36] #4  void Foam::fvMeshBalance::correctBoundaries<Foam::GeometricField<double, Foam::pointPatchField, Foam::pointMesh> >() in /lib/x86_64-linux-gnu/libstdc++.so.6
[32] #4  void Foam::fvMeshBalance::correctBoundaries<Foam::GeometricField<double, Foam::pointPatchField, Foam::pointMesh> >() in /lib/x86_64-linux-gnu/libstdc++.so.6
[33] #4  void Foam::fvMeshBalance::correctBoundaries<Foam::GeometricField<double, Foam::pointPatchField, Foam::pointMesh> >() at ??:?
[47] #3  Foam::fvMeshBalance::distribute() in /lib/x86_64-linux-gnu/libstdc++.so.6
[35] #4  void Foam::fvMeshBalance::correctBoundaries<Foam::GeometricField<double, Foam::pointPatchField, Foam::pointMesh> >() in /lib/x86_64-linux-gnu/libstdc++.so.6
[46] #4  void Foam::fvMeshBalance::correctBoundaries<Foam::GeometricField<double, Foam::pointPatchField, Foam::pointMesh> >() at ??:?

[FoamScience]

Hi @JanGaertner
Glad to see the library in continuous use :)
And, thanks for reporting the issues.
PR #11 tracks work on this because it is important to be able to post process with LB turned on.

Could you take a look at my sampling setup and replicate yours for the tutorial damBreak2D case (maybe PR yours into the hotfix/sampling branch)? just so we're on the same page. Also try out different decomposition methods: simple/hierarchical might work for you.

FYI: I will be on holidays for the next 2 weeks so It might take some time to debug this properly

[hoangquanbk59]

Hello, Thank you for your contribution. It seems this problem is still there? I tried, but this error constantly appears:
PIMPLE: iteration 1
---->! custom error estimator !<----
---->! end error estimator !<----
Protecting 0 cells next to requested boundary patches.
Selected 2 cells for refinement out of 2852.
Refined from 2852 to 2858 cells.
Selected 0 points to split 6320.
canBalance(): Maximum imbalance = 20.9237 %
polyRefinement : reading fvMeshPolyRefiner from time "0.0325702"
2D case detected. Adding prismatic2DRefinement topology modifier
polyRefinement : reading fvMeshPolyRefiner from time "0.0325702"
2D case detected. Adding prismatic2DRefinement topology modifier
1
Maximum imbalance = 20.9237 %
polyRefinement : reading fvMeshPolyRefiner from time "0.0325702"
2D case detected. Adding prismatic2DRefinement topology modifier
polyRefinement : reading fvMeshPolyRefiner from time "0.0325702"
2D case detected. Adding prismatic2DRefinement topology modifier
Mapping the fields ...
Distributing the mesh ...
Successfully distributed mesh
[3]
[3]
[3] --> FOAM FATAL ERROR: (openfoam-2406)
[3] Cannot dereference nullptr at index 6 in range [0,6)
[3]
[3]
[3] From const T& Foam::UPtrList::at(Foam::label) const [with T = Foam::polyPatch; Foam::label = int]
[3] in file /home/hoangv/OpenFOAM-v2406/src/OpenFOAM/lnInclude/UPtrListI.H at line 161.
[3]
FOAM parallel run aborting
[3]
[3] [stack trace]
[3] =============
[5]
[5]
[5] --> FOAM FATAL ERROR: (openfoam-2406)
[5] Cannot dereference nullptr at index 6 in range [0,6)
[5]
[5]
[5] From const T& Foam::UPtrList::at(Foam::label) const [with T = Foam::polyPatch; Foam::label = int]
[5] in file /home/hoangv/OpenFOAM-v2406/src/OpenFOAM/lnInclude/UPtrListI.H at line 161.
[5]
FOAM parallel run aborting
[5]
[5] [stack trace]
[5] =============
[0]
[0]
[0] --> FOAM FATAL ERROR: (openfoam-2406)
[0] Cannot dereference nullptr at index 5 in range [0,5)
[0]
[0]
[0] From const T& Foam::UPtrList::at(Foam::label) const [with T = Foam::polyPatch; Foam::label = int]
[0] in file /home/hoangv/OpenFOAM-v2406/src/OpenFOAM/lnInclude/UPtrListI.H at line 161.
[0]
FOAM parallel run aborting

[FoamScience]

Hi, I need a little bit more info as I'm unable to reliably reproduce this issue any more:

• Which OpenFOAM version are you using?
• If possible, can I see the case? or at least the sampling dictionary

As far as I can tell, OpenFOAM v2312 just works but it seems to be related to the distribution method - see #11
You can give it a try with:

git checkout hotfix/sampling
cd tutorials/damBreak2D
< Change nProcs to at least 16 to guarantee load balancer calls >
export AMRLB_PROJECT=$(git rev-parse --show-toplevel) 
./Allclean && ./Allrun

Note: Also, tried with v2406 but this particular version introduced an IO bug that looks unrelated to AMR/LB

AGAIN:

• random works, which probably means it's something other decomposition methods must take care of
• simple/hierarchical work just fine
• scotch does not work

[2022ZM]

TScas.zip

I'm getting two errors, both after Successfully distributed mesh. This is an example above, and I guess it's probably because snaapyhexmesh uses surface.stl (constant/trisurface) as the face for mesh refinement. Maybe it's because of other mistakes, please help analyze and correct it. My version is v2112.

error1

Selected 1625 cells for refinement out of 236952.
Refined from 236952 to 241827 cells.
canBalance(): Maximum imbalance = 122.492939 %
Selecting decompositionMethod scotch [64]
Selecting decompositionConstraint polyRefinement
polyRefinement : setting constraints to preserve refinement history
polyRefinement : reading fvMeshPolyRefiner from time "3.87719e-06"
2D case detected. Adding prismatic2DRefinement topology modifier
polyRefinement : reading fvMeshPolyRefiner from time "3.87719e-06"
2D case detected. Adding prismatic2DRefinement topology modifier
1
Maximum imbalance = 122.492939 %
polyRefinement : reading fvMeshPolyRefiner from time "3.87719e-06"
2D case detected. Adding prismatic2DRefinement topology modifier
polyRefinement : reading fvMeshPolyRefiner from time "3.87719e-06"
2D case detected. Adding prismatic2DRefinement topology modifier
Mapping the fields ...
Distributing the mesh ...
Successfully distributed mesh
New max imbalance: 0.967220368%
[47] [61] 
[61] 
[61] --> FOAM FATAL ERROR: (openfoam-2112)
[61] Cannot dereference nullptr at index 10 in range [0,10)
[61] 

error2

Unrefined from 186980 to 101692 cells.
canBalance(): Maximum imbalance = 85.111907 %
Decomposition method scotch [12]
Selecting decompositionConstraint polyRefinement
polyRefinement : setting constraints to preserve refinement history
polyRefinement : reading fvMeshPolyRefiner from time "0.0001"
3D case detected. Adding polyhedralRefinement topology modifier
polyRefinement : reading fvMeshPolyRefiner from time "0.0001"
3D case detected. Adding polyhedralRefinement topology modifier
1
Maximum imbalance = 85.111907 %
polyRefinement : reading fvMeshPolyRefiner from time "0.0001"
3D case detected. Adding polyhedralRefinement topology modifier
polyRefinement : reading fvMeshPolyRefiner from time "0.0001"
3D case detected. Adding polyhedralRefinement topology modifier
Mapping the fields ...
Distributing the mesh ...
Successfully distributed mesh
New max imbalance: 0.99516186%
Distributing ...
[1] #0  Foam::error::printStack(Foam::Ostream&)[3] #0  Foam::error::printStack(Foam::Ostream&)[6] #0  Foam::error::printStack(Foam::Ostream&)[9] #0  Foam::error::printStack(Foam::Ostream&)[10] #0  Foam::error::printStack(Foam::Ostream&)[8] #0  Foam::error::printStack(Foam::Ostream&) at ??:?
[3] #1  Foam::sigSegv::sigHandler(int) at ??:?
[1] #1  Foam::sigSegv::sigHandler(int) at ??:?
[6] #1  Foam::sigSegv::sigHandler(int) at ??:?
[9] #1  Foam::sigSegv::sigHandler(int) at ??:?
[8] #1  Foam::sigSegv::sigHandler(int) at ??:?
[10] #1  Foam::sigSegv::sigHandler(int) at ??:?
[3] #2  ? at ??:?
[1] #2  ? at ??:?
[10] #2  ? in /lib/x86_64-linux-gnu/libc.so.6
[3] #3  __dynamic_cast at ??:?

[FoamScience]

Thanks for your input @2022ZM, again, change the decomposition method to hierarchical and things should work just fine.

Also, I do recommend getting rid of any local/region refinement done through snappyHexMesh and replacing it with a refinement criterion in constant/dynamicMeshDict. Could you please upload the case to a Github repository so you get the credits for preparing it. I can then properly attach a fix and maybe link to it in the docs.

[2022ZM]

First of all, thank you for your reply. It can indeed run with hierarchical, but when I set the number of cores to 60, it still reports the following error. I have uploaded the calculation examples of 8-core computing (TScas) and 60-core computing (TScas2-np60) to my repository.

Unrefined from 128224 to 112908 cells.
canBalance(): Maximum imbalance = 59.347433 %
Decomposition method hierarchical [60]
Selecting decompositionConstraint polyRefinement
polyRefinement : setting constraints to preserve refinement history
polyRefinement : reading fvMeshPolyRefiner from time "5e-05"
3D case detected. Adding polyhedralRefinement topology modifier
polyRefinement : reading fvMeshPolyRefiner from time "5e-05"
3D case detected. Adding polyhedralRefinement topology modifier
1
Maximum imbalance = 59.347433 %
polyRefinement : reading fvMeshPolyRefiner from time "5e-05"
3D case detected. Adding polyhedralRefinement topology modifier
polyRefinement : reading fvMeshPolyRefiner from time "5e-05"
3D case detected. Adding polyhedralRefinement topology modifier
Mapping the fields ...
Distributing the mesh ...
Successfully distributed mesh
New max imbalance: 0.17004995%
[44]
[44]
[44] --> FOAM FATAL ERROR: (openfoam-2112 patch=220610)
[44] Cannot dereference nullptr at index 10 in range [0,10)
[44]
[44]
[44] From const T& Foam::UPtrList::operator const [with T = Foam::polyPatch; Foam::label = int]
[44] in file /usr/lib/openfoam/openfoam2112/src/OpenFOAM/lnInclude/UPtrListI.H at line 220.
[44]
FOAM parallel run aborting

[FoamScience]

Alright, I genuinely cannot reliably reproduce this issue; so let's tackle this in a more detailed way.

I have set up a quick parameter variation here: https://github.com/FoamScience/surface-sampling-bamr-debug
Please go through the README there for instructions on how to run it, it's only a couple of commands...

you can change the endTime if you suspect segfaults will happen later in the run, but please add some processors to problem.parameters.nProcs.values in the provided config.yaml and see if it works for you.

Make sure to compile blastAMR from the hotfix/sampling branch.

When all simulations are done, you should see a CSV report file containing information about the runs and whether they were fine or not. This will help me pinpoint what exactly the problem is.

Note

I got rid of the snappyHexMesh step and replaced its refinement with a native one.

[FoamScience]

By the way, maybe increase your base mesh resolution on higher number of processors, you wouldn't want to have a processor responsible only for like 5 cells

[2022ZM]

I used the TScas2-np60 calculation example. It can be calculated normally when using 8-core processing, but there is this problem when 60-core parallel computing is performed. The basic grid I refined has the same problem, so For testing purposes I reduced the grid size a bit to facilitate quick testing.

You can try the TScas2-np60 calculation example. The version I used is 2112. If there are no problems when you use this calculation example, it may be a version problem. I will change the version and test it.
TScas2-np60.zip

[FoamScience]

This is not a problem with AMR/LB and is not related to this issue or to blastAMR.
if you change your adaptiveFvMesh to staticFvMesh in constant/dynamicMeshDict, no AMR/LB will be performed and you will still get the same error. It has to do with the way your case is setup, probably wrong handling of boundary conditions...

To test if the AMR/LB works for you, you can change the number of processors and run this case.

Also, please use The Allrun scripts I provide in the tutorials as a guide, it's important to call updateMesh utility before running solvers...

[2022ZM]

you got rid of the snappyHexMesh step and replaced its refinement with a native one.
But there is no entrance to inlet_liquid? I wouldn't be able to simulate my computational model. I used the tscas（https://github.com/FoamScience/surface-sampling-bamr-debug/tree/main/tscas）you set, and the following problem occurred at about 0.000595s，it will stop continuing calculations. When I changed the https://github.com/FoamScience/surface-sampling-bamr-debug/blob/main/tscas/constant/dynamicMeshDict you modified to the https://github.com/2022ZM/tscas/blob/main/dynamicMeshDict , I can perform normal calculations. And TScas can be calculated normally with 8 processors. I don't know why it can't be calculated with 60 or more processors. I also understand that it can be calculated without snappyhexmesh to generate a grid, but my calculation grid needs to be processed by snappyhexmesh. I also uploaded the results of the 60-processor calculation to the library, which contains all logs https://github.com/2022ZM/tscas/blob/main/TScas-np60-debug.zip.

Courant Number mean: 0.00051028049 max: 0.0040564591
Interface Courant Number mean: 0 max: 0
Time = 0.000595

PIMPLE: iteration 1
Protecting 0 cells next to requested boundary patches.