【Hackathon 8th No.13】Domino 论文复现 #1093

xiaoyewww · 2025-03-04T17:00:51Z

PR types

New Features

PR changes

Others

Describe

support domino

paddle-bot · 2025-03-04T17:00:55Z

Thanks for your contribution!

xiaoyewww · 2025-03-04T17:06:18Z

复现Domino有两个问题：

目前只复现了模型内容，另外还有前处理和后处理部分，需要的CPU配置和内存非常高，aistudio上无法跑通
官方没有提供预训练权重

luotao1 · 2025-03-05T12:02:40Z

请先提交 RFC 设计文档

wangguan1995 · 2025-03-10T11:27:52Z

不好意思这里任务描述有误，需要改为【推理】和【训练】

luotao1 · 2025-03-11T02:32:20Z

@wangguan1995 可以提PR修改下任务描述 https://github.com/PaddlePaddle/community/blob/master/hackathon/hackathon_8th/%E3%80%90Hackathon_8th%E3%80%91%E4%B8%AA%E4%BA%BA%E6%8C%91%E6%88%98%E8%B5%9B%E2%80%94%E5%A5%97%E4%BB%B6%E5%BC%80%E5%8F%91%E4%BB%BB%E5%8A%A1%E5%90%88%E9%9B%86.md#no13-domino-%E8%AE%BA%E6%96%87%E5%A4%8D%E7%8E%B0

xiaoyewww · 2025-03-15T15:49:56Z

@wangguan1995 目前模型已经能正常训练，精度通过Padiff验证通过。训练代码存在随机性，每个step前处理数据没法通过随机数种子固定。
目前还有需要验证的任务有：
（1）目前缺少数据集，仅通过一个样本进行训练，50个epoch loss正常下降
（2）推理代码已经适配，但前处理部分仍需进行点云处理，aistudio上无法处理，这部分需要继续验证

wangguan1995 · 2025-04-03T08:20:49Z

目前代码仓库存在大量相对路径
下载脚本目前有一些问题（aws自己的问题，先标记在文档里）
run_1作为验证训练的数据，10个epoch的torch对比日志贴在这里
需要做的是前向loss 1e-5级别的对齐

xiaoyewww · 2025-04-03T15:20:16Z

前10 个epoch日志：

torch

Device cuda:0, batch processed: 1, loss volume: 0.23092692             , loss surface: 0.06465874, loss integral: 0.00000000, loss surface area: 0.00293629
 Device cuda:0,  batch: 1, loss norm: 0.26472443
Device cuda:0 LOSS train 0.26472443 valid 0.19338508 Current lr 0.001Integral factor 0
Device cuda:0, Best val loss 0.05669114366173744, Time taken 118.63378143310547

Device cuda:0, epoch 1:
Device cuda:0, batch processed: 1, loss volume: 0.17430124             , loss surface: 0.03637335, loss integral: 0.00000000, loss surface area: 0.00179438
 Device cuda:0,  batch: 1, loss norm: 0.19338511
Device cuda:0 LOSS train 0.19338511 valid 0.11204524 Current lr 0.001Integral factor 0
Device cuda:0, Best val loss 0.05669114366173744, Time taken 114.84539103507996

Device cuda:0, epoch 2:
Device cuda:0, batch processed: 1, loss volume: 0.09825166             , loss surface: 0.02525518, loss integral: 0.00000000, loss surface area: 0.00233155
 Device cuda:0,  batch: 1, loss norm: 0.11204502
Device cuda:0 LOSS train 0.11204502 valid 0.10328176 Current lr 0.001Integral factor 0
Device cuda:0, Best val loss 0.05669114366173744, Time taken 115.87338018417358

Device cuda:0, epoch 3:
Device cuda:0, batch processed: 1, loss volume: 0.09899043             , loss surface: 0.00795730, loss integral: 0.00000000, loss surface area: 0.00062173
 Device cuda:0,  batch: 1, loss norm: 0.10327994
Device cuda:0 LOSS train 0.10327994 valid 0.05417285 Current lr 0.001Integral factor 0
Device cuda:0, Best val loss 0.05417285114526749, Time taken 116.46289348602295

Device cuda:0, epoch 4:
Device cuda:0, batch processed: 1, loss volume: 0.04868028             , loss surface: 0.01017947, loss integral: 0.00000000, loss surface area: 0.00080577
 Device cuda:0,  batch: 1, loss norm: 0.05417290
Device cuda:0 LOSS train 0.05417290 valid 0.08227389 Current lr 0.001Integral factor 0
Device cuda:0, Best val loss 0.05417285114526749, Time taken 115.91964483261108

Device cuda:0, epoch 5:
Device cuda:0, batch processed: 1, loss volume: 0.07733711             , loss surface: 0.00911645, loss integral: 0.00000000, loss surface area: 0.00075788
 Device cuda:0,  batch: 1, loss norm: 0.08227427
Device cuda:0 LOSS train 0.08227427 valid 0.08577856 Current lr 0.001Integral factor 0
Device cuda:0, Best val loss 0.05417285114526749, Time taken 115.41759729385376

Device cuda:0, epoch 6:
Device cuda:0, batch processed: 1, loss volume: 0.08035985             , loss surface: 0.01013019, loss integral: 0.00000000, loss surface area: 0.00070755
 Device cuda:0,  batch: 1, loss norm: 0.08577872
Device cuda:0 LOSS train 0.08577872 valid 0.06831404 Current lr 0.001Integral factor 0
Device cuda:0, Best val loss 0.05417285114526749, Time taken 115.05477333068848

Device cuda:0, epoch 7:
Device cuda:0, batch processed: 1, loss volume: 0.06417362             , loss surface: 0.00766032, loss integral: 0.00000000, loss surface area: 0.00062155
 Device cuda:0,  batch: 1, loss norm: 0.06831456
Device cuda:0 LOSS train 0.06831456 valid 0.04400067 Current lr 0.001Integral factor 0
Device cuda:0, Best val loss 0.044000666588544846, Time taken 116.03830194473267

Device cuda:0, epoch 8:
Device cuda:0, batch processed: 1, loss volume: 0.04010706             , loss surface: 0.00724047, loss integral: 0.00000000, loss surface area: 0.00054595
 Device cuda:0,  batch: 1, loss norm: 0.04400026
Device cuda:0 LOSS train 0.04400026 valid 0.04530785 Current lr 0.001Integral factor 0
Device cuda:0, Best val loss 0.044000666588544846, Time taken 115.86109900474548

Device cuda:0, epoch 9:
Device cuda:0, batch processed: 1, loss volume: 0.04175998             , loss surface: 0.00659478, loss integral: 0.00000000, loss surface area: 0.00050289
 Device cuda:0,  batch: 1, loss norm: 0.04530882
Device cuda:0 LOSS train 0.04530882 valid 0.06185470 Current lr 0.001Integral factor 0
Device cuda:0, Best val loss 0.044000666588544846, Time taken 115.73935866355896

Device cuda:0, epoch 10:
Device cuda:0, batch processed: 1, loss volume: 0.05804047             , loss surface: 0.00707586, loss integral: 0.00000000, loss surface area: 0.00055142
 Device cuda:0,  batch: 1, loss norm: 0.06185411
Device cuda:0 LOSS train 0.06185411 valid 0.04006197 Current lr 0.001Integral factor 0
Device cuda:0, Best val loss 0.04006196931004524, Time taken 116.31600141525269

paddle:

Device gpu:0, batch processed: 1, loss volume: 0.23092692             , loss surface: 0.06465873, loss integral: 0.00000000, loss surface area: 0.00293629
 Device gpu:0,  batch: 1, loss norm: 0.26472443
Loss/train: 0.2647244334220886/1
Device gpu:0 LOSS train 0.26472443 valid 0.19338842 Current lr 0.001Integral factor 0
Device gpu:0, Best val loss 0.05531581491231918, Time taken 118.70474576950073

Device gpu:0, epoch 1:
Device gpu:0, batch processed: 1, loss volume: 0.17430471             , loss surface: 0.03637305, loss integral: 0.00000000, loss surface area: 0.00179433
 Device gpu:0,  batch: 1, loss norm: 0.19338840
Loss/train: 0.19338840246200562/2
Device gpu:0 LOSS train 0.19338840 valid 0.11218615 Current lr 0.001Integral factor 0
Device gpu:0, Best val loss 0.05531581491231918, Time taken 115.69161486625671

Device gpu:0, epoch 2:
Device gpu:0, batch processed: 1, loss volume: 0.09839579             , loss surface: 0.02524963, loss integral: 0.00000000, loss surface area: 0.00233097
 Device gpu:0,  batch: 1, loss norm: 0.11218609
Loss/train: 0.11218608915805817/3
Device gpu:0 LOSS train 0.11218609 valid 0.10281664 Current lr 0.001Integral factor 0
Device gpu:0, Best val loss 0.05531581491231918, Time taken 115.92429780960083

Device gpu:0, epoch 3:
Device gpu:0, batch processed: 1, loss volume: 0.09853126             , loss surface: 0.00795383, loss integral: 0.00000000, loss surface area: 0.00062137
 Device gpu:0,  batch: 1, loss norm: 0.10281885
Loss/train: 0.10281885415315628/4
Device gpu:0 LOSS train 0.10281885 valid 0.05420898 Current lr 0.001Integral factor 0

Device gpu:0, Best val loss 0.05420897901058197, Time taken 116.74323606491089

Device gpu:0, epoch 4:
Device gpu:0, batch processed: 1, loss volume: 0.04871760             , loss surface: 0.01017731, loss integral: 0.00000000, loss surface area: 0.00080539
 Device gpu:0,  batch: 1, loss norm: 0.05420895
Loss/train: 0.05420895293354988/5
Device gpu:0 LOSS train 0.05420895 valid 0.08210348 Current lr 0.001Integral factor 0
Device gpu:0, Best val loss 0.05420897901058197, Time taken 115.5925304889679

Device gpu:0, epoch 5:
Device gpu:0, batch processed: 1, loss volume: 0.07717736             , loss surface: 0.00909785, loss integral: 0.00000000, loss surface area: 0.00075586
 Device gpu:0,  batch: 1, loss norm: 0.08210421
Loss/train: 0.08210421353578568/6
Device gpu:0 LOSS train 0.08210421 valid 0.08545748 Current lr 0.001Integral factor 0
Device gpu:0, Best val loss 0.05420897901058197, Time taken 115.61996293067932

Device gpu:0, epoch 6:
Device gpu:0, batch processed: 1, loss volume: 0.08003174             , loss surface: 0.01014488, loss integral: 0.00000000, loss surface area: 0.00070565
 Device gpu:0,  batch: 1, loss norm: 0.08545700
Loss/train: 0.08545700460672379/7
Device gpu:0 LOSS train 0.08545700 valid 0.06784783 Current lr 0.001Integral factor 0
Device gpu:0, Best val loss 0.05420897901058197, Time taken 116.20984172821045

Device gpu:0, epoch 7:
Device gpu:0, batch processed: 1, loss volume: 0.06372426             , loss surface: 0.00762877, loss integral: 0.00000000, loss surface area: 0.00061806
 Device gpu:0,  batch: 1, loss norm: 0.06784768
Loss/train: 0.06784767657518387/8
Device gpu:0 LOSS train 0.06784768 valid 0.04360897 Current lr 0.001Integral factor 0
Device gpu:0, Best val loss 0.04360896721482277, Time taken 116.99345993995667

Device gpu:0, epoch 8:
Device gpu:0, batch processed: 1, loss volume: 0.03972780             , loss surface: 0.00721701, loss integral: 0.00000000, loss surface area: 0.00054254
 Device gpu:0,  batch: 1, loss norm: 0.04360757
Loss/train: 0.043607573956251144/9
Device gpu:0 LOSS train 0.04360757 valid 0.04554129 Current lr 0.001Integral factor 0
Device gpu:0, Best val loss 0.04360896721482277, Time taken 116.08153223991394

Device gpu:0, epoch 9:
Device gpu:0, batch processed: 1, loss volume: 0.04196901             , loss surface: 0.00663531, loss integral: 0.00000000, loss surface area: 0.00050686
 Device gpu:0,  batch: 1, loss norm: 0.04554009
Loss/train: 0.04554009437561035/10
Device gpu:0 LOSS train 0.04554009 valid 0.06155418 Current lr 0.001Integral factor 0
Device gpu:0, Best val loss 0.04360896721482277, Time taken 115.41889214515686

Device gpu:0, epoch 10:
Device gpu:0, batch processed: 1, loss volume: 0.05775697             , loss surface: 0.00704247, loss integral: 0.00000000, loss surface area: 0.00054853
 Device gpu:0,  batch: 1, loss norm: 0.06155247
Loss/train: 0.061552468687295914/11
Device gpu:0 LOSS train 0.06155247 valid 0.04058465 Current lr 0.001Integral factor 0
Device gpu:0, Best val loss 0.04058464989066124, Time taken 116.95455741882324

wangguan1995 · 2025-05-19T02:52:52Z

examples/domino/README.md

@@ -0,0 +1,43 @@
+# DoMINO: Decomposable Multi-scale Iterative Neural Operator for External Aerodynamics


ReadMe需要合入文档，可以参考：https://paddlescience-docs.readthedocs.io/zh-cn/latest/zh/development/#3

已修改，麻烦review

wangguan1995 · 2025-05-19T02:53:10Z

examples/domino/modulus/launch/logging/mlflow.py

+from typing import Literal
+from typing import Tuple
+
+import torch


需要清理相关内容

wangguan1995 · 2025-05-19T02:53:17Z

examples/domino/modulus/launch/logging/mlflow.py

+    time_string = start_time.strftime("%m/%d/%y %H:%M:%S")
+    client.set_tag(run.info.run_id, "date", time_string)
+    client.set_tag(run.info.run_id, "host", os.uname()[1])
+    if torch.cuda.is_available():


需要清理相关内容

wangguan1995 · 2025-05-19T02:53:21Z

examples/domino/modulus/launch/logging/utils.py

+
+from datetime import datetime
+
+import torch


需要清理相关内容

wangguan1995 · 2025-05-19T02:53:31Z

examples/domino/modulus/launch/utils/checkpoint.py

+    path : str
+        Path to training checkpoint
+    models : Union[paddle.nn.Layer, List[paddle.nn.Layer], None], optional
+        A single or list of PyTorch models, by default None


需要清理相关内容

wangguan1995 · 2025-05-19T02:54:06Z

examples/domino/README.md

+
+## 推理
+
+1. 修改`conf/config.yaml`路径, 原始配置文件参考[DoMINO](https://github.com/NVIDIA/modulus/blob/main/examples/cfd/external_aerodynamics/domino/src/conf/config.yaml)。


modulus相关内容代码需要进一步整理，使得文件夹的相关内容可以合入PaddleScience的arch文件夹

已整理合入arch文件，麻烦review

wangguan1995 · 2025-05-19T02:58:08Z

目前计划进行数据精度验证

5.19 ~ 5.21 进行数据处理
5.22 ~ 5.23 进行小数据集调试，并行调试
5.23 ~ 5.26 完成验证进行合入

wangguan1995

To fix

wangguan1995 · 2025-05-28T06:57:49Z

examples/domino/conf/config.yaml

@@ -0,0 +1,124 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.


Copy Right需要改为PaddlePaddle，并声明是对原始论文的复现

wangguan1995 · 2025-05-28T06:59:06Z

examples/domino/process_data.py

@@ -0,0 +1,109 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.


合入至 ppsci/data/process

wangguan1995 · 2025-05-28T06:59:42Z

examples/domino/requirements.txt

@@ -0,0 +1,6 @@
+hydra-core


合入至https://github.com/PaddlePaddle/PaddleScience/blob/develop/requirements.txt

wangguan1995 · 2025-05-28T07:01:02Z

ppsci/arch/physicsnemo/datapipes/cae/__init__.py

@@ -0,0 +1,17 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.


data pipe文件夹需要按照要求合入至ppsci/data/dataset模块

wangguan1995 · 2025-05-28T07:02:28Z

ppsci/arch/physicsnemo/distributed/manager.py

solver支持分布式并行，参考：https://github.com/PaddlePaddle/PaddleScience/blob/develop/ppsci/solver/train.py进行合并

这部分manager对代码影响不大，只用到了rank和world size，直接用paddle.dist.get_rank和get_world_size做了替换

wangguan1995 · 2025-05-28T07:02:50Z

ppsci/arch/physicsnemo/launch/utils/__init__.py

@@ -0,0 +1,18 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.


wangguan1995 · 2025-05-28T07:02:59Z

ppsci/arch/physicsnemo/launch/utils/checkpoint.py

@@ -0,0 +1,378 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.


wangguan1995 · 2025-05-28T07:04:21Z

ppsci/arch/physicsnemo/models/model.py

@@ -0,0 +1,1178 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.


需要将多个py文件合并为单个ppsci/arch/physicsnemo.py
参考：ppsci/arch/kan.py

wangguan1995 · 2025-05-28T07:06:07Z

ppsci/arch/physicsnemo/utils/sdf.py

@@ -0,0 +1,143 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.


需要合入至：ppsci/utils/ 模块

wangguan1995 · 2025-05-28T07:07:21Z

examples/domino/train.py

@@ -0,0 +1,946 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.


train 和 test需要合并

paddle-bot bot added the contributor label Mar 4, 2025

luotao1 mentioned this pull request Mar 5, 2025

【Hackathon 8th】开源贡献个人挑战赛 PaddlePaddle/Paddle#71310

Open

luotao1 added the PaddlePaddle Hackathon label Mar 5, 2025

luotao1 assigned luotao1 and wangguan1995 Mar 5, 2025

xiaoyewww force-pushed the donimo branch from b57da93 to 8cd4378 Compare March 15, 2025 15:35

wangguan1995 reviewed May 19, 2025

View reviewed changes

xiaoyewww added 4 commits May 20, 2025 23:39

feat(domino): support domino

1fb4980

feat(domino): support domino for training and test

a313b5d

feat(domino): support domino for training and test

3eea7c0

feat(domino): support domino for training and test

5d0f63e

xiaoyewww force-pushed the donimo branch from 8cd4378 to 5d0f63e Compare May 21, 2025 16:09

xiaoyewww added 2 commits May 22, 2025 22:38

feat(domino): remove something of torch

baa40d3

feat(domino): move physicsnemo to arch

f7918c7

wangguan1995 reviewed May 28, 2025

View reviewed changes

xiaoyewww added 4 commits May 29, 2025 02:00

feat(domino): refactor physicsnemo

905be9a

feat(domino): refactor physicsnemo

7074ccf

feat(domino): refactor physicsnemo

5df9b6d

feat(domino): refactor physicsnemo

385a128

		@@ -0,0 +1,43 @@
		# DoMINO: Decomposable Multi-scale Iterative Neural Operator for External Aerodynamics


		## 推理

		1. 修改`conf/config.yaml`路径, 原始配置文件参考[DoMINO](https://github.com/NVIDIA/modulus/blob/main/examples/cfd/external_aerodynamics/domino/src/conf/config.yaml)。

		@@ -0,0 +1,124 @@
		# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.

		@@ -0,0 +1,109 @@
		# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.

		@@ -0,0 +1,17 @@
		# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.

		@@ -0,0 +1,18 @@
		# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.

		@@ -0,0 +1,378 @@
		# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.

		@@ -0,0 +1,1178 @@
		# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.

		@@ -0,0 +1,143 @@
		# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.

		@@ -0,0 +1,946 @@
		# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.

【Hackathon 8th No.13】Domino 论文复现 #1093

Are you sure you want to change the base?

【Hackathon 8th No.13】Domino 论文复现 #1093

Uh oh!

Conversation

xiaoyewww commented Mar 4, 2025

PR types

PR changes

Describe

Uh oh!

paddle-bot bot commented Mar 4, 2025

Uh oh!

xiaoyewww commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luotao1 commented Mar 5, 2025

Uh oh!

wangguan1995 commented Mar 10, 2025

Uh oh!

luotao1 commented Mar 11, 2025

Uh oh!

xiaoyewww commented Mar 15, 2025

Uh oh!

wangguan1995 commented Apr 3, 2025

Uh oh!

xiaoyewww commented Apr 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xiaoyewww May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangguan1995 commented May 19, 2025

Uh oh!

wangguan1995 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xiaoyewww commented Mar 4, 2025 •

edited

Loading

xiaoyewww May 22, 2025 •

edited

Loading