Merge pull request #1019 from parthenon-hpc-lab/jmm/output-nodes-2

[breaking] output more topological elements

CHANGELOG.md

@@ -37,6 +37,7 @@ Date: 2024-03-21
 - [[PR 996]](https://github.com/parthenon-hpc-lab/parthenon/pull/996) Remove dynamic allocations from swarm particle creation
 
 ### Changed (changing behavior/API/variables/...)
+- [[PR1019](https://github.com/parthenon-hpc-lab/parthenon/pull/1019) Enable output for non-cell-centered variables
 - [[PR 973]](https://github.com/parthenon-hpc-lab/parthenon/pull/973) Multigrid performance upgrades
 
 ### Fixed (not changing behavior/API/variables/...)
@@ -55,6 +56,7 @@ Date: 2024-03-21
 - [[PR 982]](https://github.com/parthenon-hpc-lab/parthenon/pull/982) add some gut check testing for parthenon-VIBE
 
 ### Incompatibilities (i.e. breaking changes)
+- [[PR1019](https://github.com/parthenon-hpc-lab/parthenon/pull/1019) Remove support for file formats < 3
 - [[PR 987]](https://github.com/parthenon-hpc-lab/parthenon/pull/987) Change the API for what was IterativeTasks
 - [[PR 974]](https://github.com/parthenon-hpc-lab/parthenon/pull/974) Change GetParentPointer to always return T*
 - [[PR 996]](https://github.com/parthenon-hpc-lab/parthenon/pull/996) Remove dynamic allocations from swarm particle creation

README.md

@@ -13,11 +13,12 @@ Parthenon -- a performance portable block-structured adaptive mesh refinement fr
 * High performance by
   * device first/device resident approach (work data only in device memory to prevent expensive transfers between host and device)
   * transparent packing of data across blocks (to reduce/hide kernel launch latency)
-  * direct device-to-device communication via asynchronous, one-sided  MPI communication
+  * direct device-to-device communication via asynchronous MPI communication
 * Intermediate abstraction layer to hide complexity of device kernel launches
 * Flexible, plug-in package system
 * Abstract variables controlled via metadata flags
 * Support for particles
+* Support for cell-, node-, face-, and edge-centered fields
 * Multi-stage drivers/integrators with support for task-based parallelism
 
 # Community

doc/sphinx/index.rst

@@ -15,11 +15,12 @@ Key Features
 
 * Device first/device resident approach (work data only in device memory to prevent expensive transfers between host and device)
 * Transparent packing of data across blocks (to reduce/hide kernel launch latency)
-* Direct device-to-device communication via asynchronous, one-sided  MPI communication
+* Direct device-to-device communication via asynchronous MPI communication
 * Intermediate abstraction layer to hide complexity of device kernel launches
 * Flexible, plug-in package system
 * Abstract variables controlled via metadata flags
 * Support for particles
+* Support for cell-, node-, face-, and edge-centered fields
 * Multi-stage drivers/integrators with support for task-based parallelism
 
 Community

doc/sphinx/src/outputs.rst

@@ -398,6 +398,11 @@ capable of opening and visualizing Parthenon graphics dumps. In both
 cases, the ``.xdmf`` files should be opened. In ParaView, select the
 “XDMF Reader” when prompted.
 
+.. warning::
+   Currently parthenon face- and edge- centered data is not supported
+   for ParaView and VisIt. However, our python tooling does support
+   all mesh locations.
+
 Preparing outputs for ``yt``
 ----------------------------
 

scripts/python/packages/parthenon_tools/parthenon_tools/movie2d.py

@@ -21,7 +21,12 @@
 
 from argparse import ArgumentParser
 from pathlib import Path
-from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
+from concurrent.futures import (
+    ThreadPoolExecutor,
+    ProcessPoolExecutor,
+    wait,
+    ALL_COMPLETED,
+)
 
 import matplotlib as mpl
 import matplotlib.pyplot as plt
@@ -208,17 +213,26 @@ def plot_dump(
         ye = yf
 
     # get tensor components
-    if len(q.shape) > 6:
-        raise ValueError(
-            "Tensor rank is higher than I can handle. "
-            + "Please revise the movie2d script"
-        )
-    if len(q.shape) == 6:
-        q = q[:, components[0], components[1], 0, :, :]
-    if len(q.shape) == 5:
-        q = q[:, components[-1], 0, :, :]
-    if len(q.shape) == 4:
-        q = q[:, 0, :, :]
+    ntensors = len(q.shape[1:-3])
+    if components:
+        if len(components) != ntensors:
+            raise ValueError(
+                "Tensor rank not the same as number of specified components: {}, {}, {}".format(
+                    len(components), ntensors, q.shape
+                )
+            )
+        # The first index of q is block index. Here we walk through
+        # the tensor components, slowest-first and, iteratively, fix
+        # q's slowest moving non-block index to the fixed tensor
+        # component. Then we move to the next index.
+        for c in components:
+            if c > (q.shape[1] - 1):
+                raise ValueError(
+                    "Component {} out of bounds. Shape = {}".format(c, q.shape)
+                )
+            q = q[:, c]
+    # move to 2d
+    q = q[..., 0, :, :]
 
     fig = plt.figure()
     p = fig.add_subplot(111, aspect=1)
@@ -299,15 +313,16 @@ def plot_dump(
     args.output_directory.mkdir(0o755, True, True)
     logger.info(f"Total files to process: {len(args.files)}")
 
-    components = [0, 0]
+    components = []
     if args.tc is not None:
         components = args.tc
     if args.vc is not None:
-        components = [0, args.vc]
+        components = [args.vc]
     do_swarm = args.swarm is not None
 
     _x = ProcessPoolExecutor if args.worker_type == "process" else ThreadPoolExecutor
     with _x(max_workers=args.workers) as pool:
+        futures = []
         for frame_id, file_name in enumerate(args.files):
             data = phdf(file_name)
 
@@ -376,40 +391,49 @@ def plot_dump(
             # NOTE: After doing 5 test on different precision, keeping 2 looks more promising
             current_time = format(round(data.Time, 2), ".2f")
             if args.debug_plot:
-                pool.submit(
-                    plot_dump,
-                    data.xg,
-                    data.yg,
-                    q,
-                    current_time,
-                    output_file,
-                    True,
-                    data.gid,
-                    data.xig,
-                    data.yig,
-                    data.xeg,
-                    data.yeg,
-                    components,
-                    swarmx,
-                    swarmy,
-                    swarmcolor,
-                    particlesize,
+                futures.append(
+                    pool.submit(
+                        plot_dump,
+                        data.xg,
+                        data.yg,
+                        q,
+                        current_time,
+                        output_file,
+                        True,
+                        data.gid,
+                        data.xig,
+                        data.yig,
+                        data.xeg,
+                        data.yeg,
+                        components,
+                        swarmx,
+                        swarmy,
+                        swarmcolor,
+                        particlesize,
+                    )
                 )
             else:
-                pool.submit(
-                    plot_dump,
-                    data.xng,
-                    data.yng,
-                    q,
-                    current_time,
-                    output_file,
-                    True,
-                    components=components,
-                    swarmx=swarmx,
-                    swarmy=swarmy,
-                    swarmcolor=swarmcolor,
-                    particlesize=particlesize,
+                futures.append(
+                    pool.submit(
+                        plot_dump,
+                        data.xng,
+                        data.yng,
+                        q,
+                        current_time,
+                        output_file,
+                        True,
+                        components=components,
+                        swarmx=swarmx,
+                        swarmy=swarmy,
+                        swarmcolor=swarmcolor,
+                        particlesize=particlesize,
+                    )
                 )
+        wait(futures, return_when=ALL_COMPLETED)
+        for future in futures:
+            exception = future.exception()
+            if exception:
+                raise exception
 
     if not ERROR_FLAG:
         logger.info("All frames produced.")