Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Alternative for zero-copy foreign share iobufs #23999

Closed
wants to merge 2 commits into from
Closed

[WIP] Alternative for zero-copy foreign share iobufs #23999

wants to merge 2 commits into from
+47 −35

Conversation

ballard26
Copy link
Contributor

This PR is an alternative to atomically reference counting in redpanda-data/seastar#149 to remove the need to copy data when moving iobufs across shards.

A few additional steps are needed to finalize this solution;

  • More iobuf cross-shard checks, better doc #23263 will need to be modified to allow an iobuf to be "re-homed" once to a different shard when it has been made foreign. This PR is what will prevent accidental misuse of the iobuf that could result in segfaults.
  • foreign_data_t and therefore the variant type could be removed from record_batch_reader
  • It's possible to lazily transform iobuf/record_batch into their "foreign" variants to avoid the need to iterate over all io_fragments in either immediately. This could help performance if not all io_fragments are always accessed.
  • Coarser grain deleters could be used to reduce the number of cross shard calls.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.2.x
  • v24.1.x
  • v23.3.x

Release Notes

  • none

@travisdowns
Copy link
Member

/dt

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Nov 5, 2024 •
edited
Loading

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57590#0192fcb1-8360-4a4b-83e9-174fb178162e:

"rptest.tests.data_transforms_test.DataTransformsLeadershipChangingTest.test_leadership_changing_randomly"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.NONE"
"rptest.tests.data_transforms_test.DataTransformsTest.test_consume_from_offset.offset=.9223372036854775807"
"rptest.tests.data_transforms_test.DataTransformsTest.test_identity.transactional=True.wait_running=False"
"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_tunable_configs"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57590#0192fcb1-8359-4fb6-a406-390b0b7e22bc:

"rptest.tests.data_transforms_test.DataTransformsMultipleOutputTopicsTest.test_multiple_output_topics"
"rptest.tests.data_transforms_test.DataTransformsTest.test_consume_from_offset.offset=-1"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.PRODUCER"
"rptest.tests.data_transforms_test.DataTransformsTest.test_identity.transactional=True.wait_running=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57590#0192fca2-fa4a-4ede-91d5-00afcf675e20:

"rptest.tests.data_transforms_test.DataTransformsTest.test_identity.transactional=False.wait_running=False"
"rptest.tests.data_transforms_test.DataTransformsTest.test_consume_from_offset.offset=.0"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.GZIP"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.SNAPPY"
"rptest.tests.data_transforms_test.DataTransformsLoggingMetricsTest.test_logger_metrics_values"
"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_otel"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57590#0192fcb1-835c-4603-b5c9-86fde8dc31dc:

"rptest.tests.data_transforms_test.DataTransformsTest.test_consume_from_offset.offset=.0"
"rptest.tests.data_transforms_test.DataTransformsTest.test_identity.transactional=False.wait_running=False"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.GZIP"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.SNAPPY"
"rptest.tests.data_transforms_test.DataTransformsLoggingMetricsTest.test_logger_metrics_values"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57590#0192fca2-fa48-4fc5-8e56-0ef2f921de65:

"rptest.tests.data_transforms_test.DataTransformsMultipleOutputTopicsTest.test_multiple_output_topics"
"rptest.tests.data_transforms_test.DataTransformsTest.test_consume_from_offset.offset=-1"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.PRODUCER"
"rptest.tests.data_transforms_test.DataTransformsTest.test_identity.transactional=True.wait_running=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57590#0192fcb1-835e-4b8a-a639-cbe39636a9b6:

"rptest.tests.data_transforms_test.DataTransformsTest.test_consume_from_offset.offset=.1730816646947"
"rptest.tests.data_transforms_test.DataTransformsChainingTest.test_multiple_transforms_chained_together"
"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.LZ4"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.ZSTD"
"rptest.tests.data_transforms_test.DataTransformsTest.test_identity.transactional=False.wait_running=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57590#0192fca2-fa43-419b-ae8b-b42bc495cf16:

"rptest.tests.data_transforms_test.DataTransformsTest.test_consume_from_offset.offset=.1730815790059"
"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.ZSTD"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.LZ4"
"rptest.tests.data_transforms_test.DataTransformsChainingTest.test_multiple_transforms_chained_together"
"rptest.tests.data_transforms_test.DataTransformsTest.test_identity.transactional=False.wait_running=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57590#0192fca2-fa46-4910-b37f-1adeb68cec95:

"rptest.tests.data_transforms_test.DataTransformsLeadershipChangingTest.test_leadership_changing_randomly"
"rptest.tests.data_transforms_test.DataTransformsTest.test_consume_from_offset.offset=.9223372036854775807"
"rptest.tests.data_transforms_test.DataTransformsTest.test_compression.compression_type=CompressionTypes.NONE"
"rptest.tests.data_transforms_test.DataTransformsTest.test_identity.transactional=True.wait_running=False"

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Nov 5, 2024 •
edited
Loading

Retry command for Build#57590

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/data_transforms_test.py::DataTransformsLeadershipChangingTest.test_leadership_changing_randomly
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_compression@{"compression_type":"none"}
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_consume_from_offset@{"offset":"+9223372036854775807"}
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_identity@{"transactional":true,"wait_running":false}
tests/rptest/tests/data_transforms_test.py::DataTransformsLoggingTest.test_tunable_configs
tests/rptest/tests/data_transforms_test.py::DataTransformsMultipleOutputTopicsTest.test_multiple_output_topics
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_consume_from_offset@{"offset":"-1"}
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_compression@{"compression_type":"producer"}
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_identity@{"transactional":true,"wait_running":true}
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_identity@{"transactional":false,"wait_running":false}
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_consume_from_offset@{"offset":"+0"}
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_compression@{"compression_type":"gzip"}
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_compression@{"compression_type":"snappy"}
tests/rptest/tests/data_transforms_test.py::DataTransformsLoggingMetricsTest.test_logger_metrics_values
tests/rptest/tests/data_transforms_test.py::DataTransformsLoggingTest.test_logs_otel
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_consume_from_offset@{"offset":"@1730816646947"}
tests/rptest/tests/data_transforms_test.py::DataTransformsChainingTest.test_multiple_transforms_chained_together
tests/rptest/tests/data_transforms_test.py::DataTransformsLoggingTest.test_logs_volume
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_compression@{"compression_type":"lz4"}
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_compression@{"compression_type":"zstd"}
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_identity@{"transactional":false,"wait_running":true}
tests/rptest/tests/data_transforms_test.py::DataTransformsTest.test_consume_from_offset@{"offset":"@1730815790059"}

@vbotbuildovich
Copy link
Collaborator

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/57590#0192fca2-fa46-4910-b37f-1adeb68cec95

@ballard26
model: properly set owner_shard for record_batch in transformed_data:…
2ff1b48 
…:make_batch

The owner_shard is set to std::nullopt by default and often implicitly
set to a shard when the record_batch is copied. This change ensures that
it'll be set even if the record_batch isn't copied.
@ballard26
Copy link
Contributor Author

We've decided to go with #24047 instead of this PR.

@ballard26 ballard26 closed this Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/redpanda
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ballard26 @travisdowns @vbotbuildovich