Manifests migration #5575

nccatoni · 2025-10-22T12:28:12Z

Motivation

Changes

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

If PR title starts with [<language>], double-check that only <language> is impacted by the change
No system-tests internal is modified. Otherwise, I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added (or removed)?
- Get a review from R&P team

github-actions · 2025-10-22T12:28:50Z

CODEOWNERS have been resolved as:

migration.py                                                            @DataDog/system-tests-core
tests/test_the_test/manifests/manifests_parser_test/java.yml            @DataDog/system-tests-core
tests/test_the_test/manifests/manifests_parser_test/python.yml          @DataDog/system-tests-core
utils/manifest/__init__.py                                              @DataDog/system-tests-core
utils/manifest/core.py                                                  @DataDog/system-tests-core
utils/manifest/declaration.py                                           @DataDog/system-tests-core
utils/manifest/parser.py                                                @DataDog/system-tests-core
utils/manifest/rule.py                                                  @DataDog/system-tests-core
utils/manifest/schema.json                                              @DataDog/system-tests-core
utils/manifest/validate.py                                              @DataDog/system-tests-core
conftest.py                                                             @DataDog/system-tests-core
format.sh                                                               @DataDog/system-tests-core
manifests/agent.yml                                                     @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/cpp_httpd.yml                                                 @DataDog/system-tests-core
manifests/cpp_nginx.yml                                                 @DataDog/system-tests-core
manifests/dd_apm_inject.yml                                             @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/k8s_cluster_agent.yml                                         @DataDog/system-tests-core
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/python_lambda.yml                                             @DataDog/system-tests-core
manifests/python_otel.yml                                               @DataDog/system-tests-core
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
tests/test_the_test/manifests/manifests_agent_edit/agent.yml            @DataDog/system-tests-core
tests/test_the_test/manifests/manifests_agent_edit/python.yml           @DataDog/system-tests-core
tests/test_the_test/manifests/manifests_python_edit/agent.yml           @DataDog/system-tests-core
tests/test_the_test/manifests/manifests_python_edit/python.yml          @DataDog/system-tests-core
tests/test_the_test/manifests/manifests_ref/agent.yml                   @DataDog/system-tests-core
tests/test_the_test/manifests/manifests_ref/python.yml                  @DataDog/system-tests-core
tests/test_the_test/test_compute_libraries_and_scenarios.py             @DataDog/system-tests-core
tests/test_the_test/test_manifest.py                                    @DataDog/system-tests-core
utils/_decorators.py                                                    @DataDog/system-tests-core
utils/scripts/compute_libraries_and_scenarios.py                        @DataDog/system-tests-core
manifests/parser/__init__.py                                            @DataDog/system-tests-core
manifests/parser/core.py                                                @DataDog/system-tests-core
manifests/parser/schema.json                                            @DataDog/system-tests-core
tests/test_the_test/test_existing_versions.py                           @DataDog/system-tests-core

tests/test_the_test/manifests/manifests_agent_edit/python.yml

tests/test_the_test/test_compute_libraries_and_scenarios.py

cbeauchesne · 2025-11-21T14:48:15Z

tests/test_the_test/test_existing_versions.py

@@ -1,129 +0,0 @@
-import os


Why this file has been removed ?

cbeauchesne · 2025-11-21T14:57:19Z

utils/manifest/_internal/declaration.py

+        self.parse_declaration()
+
+    @staticmethod
+    def fix_separator(version: str) -> str:


I feel like we should be more strict with the content of manifest, less complexity, and cleaner data.

Do we have lot of use case to fix ?

I agree but we won't be able to fix the versioning in the libraries themselves and having a different version syntax in the manifests compared to the rest of the lib repo could be confusing

cbeauchesne · 2025-11-21T15:00:34Z

utils/manifest/_internal/rule.py

+from .types import ManifestData, Condition, SkipDeclaration
+
+
+def match_condition(


This function should a method of Condition, no ?

It could but it is never used outside of match_rule

cbeauchesne · 2025-11-21T15:05:53Z

utils/manifest/_internal/rule.py

+    path = rule.split("/")
+    path = [part for part in path if part]
+    rest = rule.split("::")
+    rule_elements = [*path[:-1], path[-1].split("::")[0], *rest[1:]]


Can be done with rule_elements = Path(rule.replace("::", "/")).parts

pathlib.Path is a great tool to manipulate paths.

rule_elements = rule.replace("::", "/").split("/") would also work no ?

cbeauchesne · 2025-11-21T15:08:53Z

utils/manifest/_internal/rule.py

+    if len(rule_elements) > len(nodeid_elements):
+        return False
+    return all(elements[0] == elements[1] for elements in zip(rule_elements, nodeid_elements, strict=False))


Suggested change

if len(rule_elements) > len(nodeid_elements):

return False

return all(elements[0] == elements[1] for elements in zip(rule_elements, nodeid_elements, strict=False))

return nodeid_elements[:len(rule_elements)] == rule_elements

cbeauchesne · 2025-11-21T15:17:12Z

utils/_decorators.py

    force_skip: bool = False,
 ):
-    if not inspect.isfunction(item) and not inspect.isclass(item) and not isinstance(item, pytest.Module):
+    if (


IIUC, this block should be isinstance(item, pytest.Item) only ?

mmmh, no, it's can be called b decorator, so it's the prototype that should be kept to pytest.Item | FunctionType | MethodType

Migration script

fe77e2f

nccatoni added 18 commits October 22, 2025 18:03

Multi file migration

393005f

Updated manifests

e497d83

Rough

7e54f73

Cleanup

93f8ec6

Simplified manifests validation

b4fcc43

Fixes

0ec3121

Fixes

7b2e661

Manifest validation

f1f7001

Updating test selection testing

b76dab3

Fixing sorting

94573f9

New manifest tests

5fa7be7

Fixes

7170eda

Manifest update

655666c

Adding support for references (nodejs)

5267a67

resolve merge conflict

9432a8c

Merge branch 'main' into nccatoni/manifest-migration

a844a76

Manifest update plus migration fix

43c7d99

Conftest dump fix

5c93321

nccatoni changed the title ~~Migration script~~ Manifests migration Nov 3, 2025

nccatoni force-pushed the nccatoni/manifest-migration branch from 3addd02 to 84b9daa Compare November 3, 2025 16:17

To be reverted (skip manifest comparison)

aab6bb4

nccatoni force-pushed the nccatoni/manifest-migration branch from 84b9daa to aab6bb4 Compare November 3, 2025 16:20

nccatoni added 6 commits November 4, 2025 13:19

Robust rule matching

779d1c0

Version ordering check

146cccb

More robust inline version parsing

6865ee1

Merge branch 'main' into nccatoni/manifest-migration

a0fbc98

Deactivating test selection testing

5aa8522

Format

8a5dc29

Update test

9e057fb

nccatoni force-pushed the nccatoni/manifest-migration branch from 7a87ca7 to e3c5f7a Compare November 19, 2025 12:28

Test

46e60b9

nccatoni force-pushed the nccatoni/manifest-migration branch from e3c5f7a to 46e60b9 Compare November 19, 2025 12:31

nccatoni added 8 commits November 19, 2025 13:33

Fix for gitlab ci

d2ded64

Parser refactoring plus types improvements

bd8817b

Static analyzer

ec088be

Update export

1b38c36

Moving item marking to the right place plus type improvement

e70c878

Docstring update

1a7714c

Improved return type for the public api

a43bb2a

Adding _

3e9fcbb

nccatoni force-pushed the nccatoni/manifest-migration branch from 6647ba8 to 3e9fcbb Compare November 20, 2025 15:57

nccatoni added 5 commits November 21, 2025 11:41

Module organization

afb7809

Fix, documentation and better error handling

7871cac

More docs

369008f

More docs

3365454

Typo

4250351