Improve performance with nested PacketFields

[alxroyer-thales]

Draft PR for the following issues:

• Performance issues with nested PacketFields #4705
• Force cache reset as soon as the payload or a packet field is modified #4706
• [enhancement] Enable clear_cache() to spread upward in the packet tree #4707
• fix for Packet.parent references missing for packet fields with default instantiations.

Initiated as a draft PR for preliminary discussions before an official delivery (not squashed yet).

To be discussed:

• Changes currently based on v2.6.1 => shall I align on master? or useful to deliver a fix for a version before?
• Packet.explicit flag.
• Packet.raw_packet_cache_fields removed.

Checklist:

• If you are new to Scapy: I have checked CONTRIBUTING.md (esp. section submitting-pull-requests)
• I squashed commits belonging together
• I added unit tests or explained why they are not relevant
• I executed the regression tests (using cd test && ./run_tests or tox)
• If the PR is still not finished, please create a Draft Pull Request

[alxroyer-thales]

A couple of figures about the impact of the changes:

'perf_packet_fields.uts' times before changes:

passed 2E618233 000.02s 000) Define `NUMBER_OF_I_PER_M` and `NUMBER_OF_F_PER_I` constants, and prepare a base `TestPacket` class.
passed 9DC21D20 000.00s 001) Define the `F` final packet class.
passed B7BBB574 000.01s 002) Define the `I` intermediate packet class.
passed E12AE3F0 000.54s 003) Define the `M` main packet class.
passed 10E80BEA 000.00s 004) Build a default instance of `F`.
passed 59DBEF73 000.01s 005) Build a default instance of `I`.
failed B573553A 002.91s 006) Build a default instance of `M`.
failed A6E98C89 005.31s 007) Launch serialization from the latest instance of `M` created.
failed 4DAB5BC5 005.06s 008) Launch serialization again from the same instance of `M`.
failed 46AD5D98 005.21s 009) Update one `F` from the same instance of `M` and launch serialization again.
passed B8CC085E 001.28s 010) Parse the buffer serialized before.

After changes:

passed 2E618233 000.02s 000) Define `NUMBER_OF_I_PER_M` and `NUMBER_OF_F_PER_I` constants, and prepare a base `TestPacket` class.
passed 9DC21D20 000.00s 001) Define the `F` final packet class.
passed B7BBB574 000.01s 002) Define the `I` intermediate packet class.
passed E12AE3F0 000.50s 003) Define the `M` main packet class.
passed 10E80BEA 000.00s 004) Build a default instance of `F`.
passed 59DBEF73 000.00s 005) Build a default instance of `I`.
passed B573553A 000.53s 006) Build a default instance of `M`.
passed A6E98C89 000.13s 007) Launch serialization from the latest instance of `M` created.
passed 4DAB5BC5 000.00s 008) Launch serialization again from the same instance of `M`.
passed 46AD5D98 000.00s 009) Update one `F` from the same instance of `M` and launch serialization again.
passed B8CC085E 001.22s 010) Parse the buffer serialized before.

Biggest impacts on default instantiations (fewer copies) and serializations (cache optimizations).
(It seems parsing had already been optimized with version 2.5.0.)

Before changes:

######
## Default instantiations
######


###(006)=[failed] 006) Build a default instance of `M`.

>>> TestPacket.begin_case()
>>> m = M()
>>> TestPacket.end_case(
...     init_m=1,
...     init_i=NUMBER_OF_I_PER_M,
...     init_f=NUMBER_OF_I_PER_M * NUMBER_OF_F_PER_I,
... )
Number of M inits: 1 (expected: 1)
Number of I inits: 200 (expected: 100)
Number of F inits: 60000 (expected: 10000)
Number of M builds: 0 (expected: 0)
Number of I builds: 0 (expected: 0)
Number of F builds: 0 (expected: 0)
Total number of operations expected: 10101
Execution time: 2.912 seconds (max: 1.667)
Traceback (most recent call last):
  File "<input>", line 5, in <module>
  File "<input>", line 45, in end_case
AssertionError: Execution time FAILED: 2.912 > 1.667


######
## Serialization
######


###(007)=[failed] 007) Launch serialization from the latest instance of `M` created.

>>> TestPacket.begin_case()
>>> buf = m.build()
>>> TestPacket.end_case(
...     build_m=1,
...     build_i=NUMBER_OF_I_PER_M,
...     build_f=NUMBER_OF_I_PER_M * NUMBER_OF_F_PER_I,
... )
Number of M inits: 1 (expected: 0)
Number of I inits: 300 (expected: 0)
Number of F inits: 90000 (expected: 0)
Number of M builds: 1 (expected: 1)
Number of I builds: 100 (expected: 100)
Number of F builds: 10000 (expected: 10000)
Total number of operations expected: 10101
Execution time: 5.307 seconds (max: 1.667)
Traceback (most recent call last):
  File "<input>", line 5, in <module>
  File "<input>", line 45, in end_case
AssertionError: Execution time FAILED: 5.307 > 1.667


###(008)=[failed] 008) Launch serialization again from the same instance of `M`.

>>> TestPacket.begin_case()
>>> buf = m.build()
>>> TestPacket.end_case(
...     build_m=1,  # `m` is cached, `build()` not spreaded on `i{n}` fields (thus, no `f{n}`).
...     build_i=0,
...     build_f=0,
... )
Number of M inits: 1 (expected: 0)
Number of I inits: 300 (expected: 0)
Number of F inits: 90000 (expected: 0)
Number of M builds: 1 (expected: 1)
Number of I builds: 100 (expected: 0)
Number of F builds: 10000 (expected: 0)
Total number of operations expected: 1
Execution time: 5.055 seconds (max: 0.010)
Traceback (most recent call last):
  File "<input>", line 5, in <module>
  File "<input>", line 45, in end_case
AssertionError: Execution time FAILED: 5.055 > 0.010


###(009)=[failed] 009) Update one `F` from the same instance of `M` and launch serialization again.

>>> m.i0.f0.value += 1
>>> 
>>> TestPacket.begin_case()
>>> buf = m.build()
>>> TestPacket.end_case(
...     build_m=1,
...     build_i=NUMBER_OF_I_PER_M,  # `i0` gets rebuilts, next `i{n}` fields are just asked for their cache.
...     build_f=1 * NUMBER_OF_F_PER_I,  # `i0.f0` gets rebuilt, next `i0.f{n}` fields are just asked for their cache.
... )
Number of M inits: 1 (expected: 0)
Number of I inits: 300 (expected: 0)
Number of F inits: 90000 (expected: 0)
Number of M builds: 1 (expected: 1)
Number of I builds: 100 (expected: 100)
Number of F builds: 10000 (expected: 100)
Total number of operations expected: 201
Execution time: 5.207 seconds (max: 0.033)
Traceback (most recent call last):
  File "<input>", line 5, in <module>
  File "<input>", line 45, in end_case
AssertionError: Execution time FAILED: 5.207 > 0.033

[alxroyer-thales]

I've also tried to see how things behave when making the two constants NUMBER_OF_I_PER_M and NUMBER_OF_F_PER_I vary.

In red, before the changes, in blue, after the changes:

With that angle, we better see how the blue surface remains below the red one while NUMBER_OF_I_PER_M and NUMBER_OF_F_PER_I rise up.

[alxroyer-thales]

The code is probably rough for the moment. That's why I opened a draft PR.
I'll be glad to read through your remarks to make it better.

[alxroyer-thales]

This no_cache flag comes like an extra feature.
It should probably be documented if we keep it.

[alxroyer-thales]

I've used Sphinx syntax to document attributes, but as far as I've seen, it's not used anywhere else.
I can change that for consistency with the rest of the code.
Let me know.

[alxroyer-thales]

Please note the removal of the raw_packet_cache_fields dictionary.
To be discussed.

[alxroyer-thales]

As noted above, the meaning of the explicit attribute should probably be explained.
To be discussed.

[alxroyer-thales]

Small fix on list copies by the way.

[alxroyer-thales]

Method added, called once only for the moment in do_init_cached_fields().
But it could/should possibly be called in other cases?
Perhaps a bit of factorization?

[alxroyer-thales]

In this new method, I've tried to stick with former implementations of copy() and clone_with(), and optimize things when relevant.
But I feel like further simplifications could be done.

[alxroyer-thales]

As proposed in #4707, I've kept the legacy downwards behaviour by default, and made the new upwards behaviour disabled by default.

[alxroyer-thales]

@gpotter2 Sorry for disturbing, but I can't figure out how to run unit tests and check results.

I'm on Windows, using Python 3.13.3 with venv, tox verion 4.25.0, branch master, commit 494e472.
I do:

cd test && ./run_tests

And even though I add assert False lines in tests like 'fields.uts' or 'random.uts', I keep on getting lines at the end of 'run_tests' executions like below:

  py313--non_root: OK (12.84 seconds)
  congratulations :) (15.69 seconds)

If I run 'fields.uts' or 'random.uts' individually, I can see the errors as expected.
What am I doing wrong with the 'run_tests' script?