Multiple Enhancement to Make MMENGINE walk further

[MGAMZ]

Motivation

The maintenance progress of mmengine is getting slower and slower at present. In order to enable this excellent architecture to progress further, I have introduced some upcoming compatibility improvements and some minor optimizations for it.

Modification

Packing according to PyPA

The packaging using setup.py is deprecated and is due to be removed. Now PyPA suggests to use pyproject.toml as the packaging configuration. So refactoring them now.

Upgrade yapf from 0.32.0 to 0.43.0

The original yapf is not supported in python3.13 due to the missing of lib2to3. And in order to use the latest yapf version, the repo site need to be changed, so the pre-commit-config.yaml undergo a small modification. This should only bring minor behavior difference.

Upgrade numpy from 1.2x to 2.2.x+

The numpy.compat is deprecated, using numpy.fft in tests/data/config/lazy_module_config/test_ast_transform.py and tests/test_config/test_lazy.py.

torch.compile configuration

The torch.compile receives a lot args. And in mmengine, all contents in compile from any config file will be transferred to torch.compile.
The mmengine checks several dependencies when user specifies compile in their config file, and making hasattr(config, compile) as the flag of enabling torch.compile.
However this is not exact. The compile config can include an arg disable to state if the compiler is actually working. So the mmengine needs to carefully determine if the user is actually enabling the torch.compile. This caused a small modification in mmengine/_strategy/base.py

Message Hub error caught

The message hub receives critical Tensor input and displays them. In it's procedure, the item method is called when handling values (can be lr, loss, metrics, time, epoch, etc...) When there exists some invalid values, the message simply crashes the process as there is an assertion.

Considering the user is easy to print something bad to the message hub at ANY code line, a more robust way is to should a warning and returns the value and see if the directly returned value can be successfully processed by outer functions.

Message Hub ignore torch.compile

The message hub's operation will inevitablely triger graph breaking during compile, so disabling it now.

Support for Python-Like config file containing model_warpper_constructor or model_wrapper

The current parsing of constructor_cfg does not support the following case (python-style configuration file):

optim_wrapper = dict(
    type = DeepSpeedOptimWrapper,
    optimizer = dict(type=AdamW, lr=lr, weight_decay=weight_decay),
    accumulative_counts = grad_accumulation,
    constructor = dict(type=DefaultOptimWrapperConstructor),
)

So adding the corresponding init logic:

constructor_cfg = optim_wrapper_cfg.pop('constructor', None)
if constructor_cfg is None:
    constructor_cfg = dict(type=DefaultOptimWrapperConstructor)

The similar issue exist on model_wrapper_cfg too, so there're some changes in mmengine.runner.runner.wrap_model too.

torch.load usage

The torch.load now requires a weights_only arg. By default, it should be set to False to prevent any risks.

Resume strategy

Advance dataloader to skip data that has already been trained on InfiniteSampled dataloader is non-sense. And could waste many times for data preprocessing, as the next(self.dataloader_iterator) operation actually performs all preprocessing steps.

Logic Breaking

In some scenarios where we need to make sure that all data are browsed at the same frequency after the model is trained. This modification might be insuitable.

Removal deprecation

• torch.cuda.amp is no longer used, replacing it with from torch.amp import GradScaler, and then set GradScaler = partial(amp_GradScaler, device='cuda')
• pkg_resources is no longer supported and is due to be completed abandoned in Nov. 2025. Replacing it with importlib. This causes many modifications in mmengine/utils/package_utils.py

Other fix

• Lint Issues
• Improve vis_backend error report hint.
• Make Lint Happy.

Include these PR

#1650
#1654 (partially)
#1639
#1610
#1617
#1608
（Maybe missing some PRs）

Checklist

Due to my personal effort, I am unable to check everything here.

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDetection or MMPretrain.
4. The documentation has been modified accordingly, like docstring or example tutorials.

[MGAMZ]

Thank you for your contribution. I agree that many of the changes are reasonable, and I'm willing to invest time to gradually merge this content.

However, merging so many changes at once carries significant risks (especially since the unit tests are currently failing). I hope we can submit these changes in separate PRs (for example, compile optimizations, lint fixes, strategy updates, etc.) and progressively move forward with merging them.

This may take some time, but if you're willing to do that, I'm happy to work with you to advance these changes.

Great, I am glad to. I was worrying the delicated PRs were useless as mmengine now lacks authorized maintainer.

And yes, It shall take some time as recently quite busy on my works.
I will refactore this big PR into small ones with higher quality.

[HAOCHENYE]

Thank you for your contribution. I agree that many of the changes are reasonable, and I'm willing to invest time to gradually merge this content.
However, merging so many changes at once carries significant risks (especially since the unit tests are currently failing). I hope we can submit these changes in separate PRs (for example, compile optimizations, lint fixes, strategy updates, etc.) and progressively move forward with merging them.
This may take some time, but if you're willing to do that, I'm happy to work with you to advance these changes.

Great, I am glad to. I was worrying the delicated PRs were useless as mmengine now lacks authorized maintainer

And yes, It shall take some time as recently quite busy on my works. I will refactore this big PR into small ones with higher quality.

Even though I have other commitments and cannot fulfill everyone's needs, I still hope that MMEngine can respond to users who are doing deep development.

Please trust that as long as you're willing to progressively and sustainably advance these PRs, I will eventually merge these changes into the main branch and release a new version.

[lauriebax]

Hi, we have noticed that MMLabs were not maintained very well. We have updated 7 of the repos, including MMEngine, to work with pytorch 2.8 and use pyproject.toml, along with some other modernization. We are committed to maintain our version of mmengine and this would help a lot too. Our repo has working CI pipelines and although we are a small team, we have some dedicated resources to review and test MR's like these. Would you be interested to open an MR in our fork (https://github.com/VBTI-development/onedl-mmengine)? And @HAOCHENYE maybe you can support us in maintaining our version, since the rest of mmlabs seems to be no longer maintained?

[MGAMZ]

@lauriebax Impressive.
But I'm quite busy recently, will consider your invitation soon.

[HAOCHENYE]

Hi, we have noticed that MMLabs were not maintained very well. We have updated 7 of the repos, including MMEngine, to work with pytorch 2.8 and use pyproject.toml, along with some other modernization. We are committed to maintain our version of mmengine and this would help a lot too. Our repo has working CI pipelines and although we are a small team, we have some dedicated resources to review and test MR's like these. Would you be interested to open an MR in our fork (https://github.com/VBTI-development/onedl-mmengine)? And @HAOCHENYE maybe you can support us in maintaining our version, since the rest of mmlabs seems to be no longer maintained?

I'll take some time this weekend to review what changes have been made in this PR and onedl mmengine compared to the current main branch, and see which modifications can be merged into main first. I can test these changes with mmengine or mmcv initially, and then try contacting the maintainers of other repos to see if we can move this forward.

[HAOCHENYE]

@MGAMZ

I've fixed the unit tests and lint issues to facilitate further validation of the PR content. However, the GitHub workflow isn't working yet, so manual local testing is required:

pytest tests --ignore tests/test_fileio/test_backends/test_petrel_backend.py.

After confirming all unit tests pass, you can have me review the PR. Additionally, I recommend removing lint-related code changes - let's focus on functional modifications first.

[MGAMZ]

@HAOCHENYE OK, I'm cherry picking the latest several commits from mmengine/main. There seems to be some format violations reported by pre-commit, so I'm fixing these format errors, this should not introduce any logical functional breaking.

And I'm running pytest and will update this PR according to the test results soon.

Maybe I will open a new PR which only contains functional modifications.

[HAOCHENYE]

Thanks @MGAMZ for your effort! I'll review the functional changes once all tests pass.

[MGAMZ]

torch.load issue has been resolved by #1670

is_builtin_module is resolved by #1669

Adafactor import error is included by #1660