Repository navigation
Add RC-RDMA (RDMA CM, v3-wire) S3 client/listener and AWS S3 demo integration - #207
Open
surya-everpure wants to merge 3 commits into
Open
surya-everpure wants to merge 3 commits into
surya-everpure wants to merge 3 commits into
Conversation
surya-everpure
marked this pull request as ready for review
October 14, 2026 10:44
surya-everpure
requested review from
akolliasAMD,
derobins,
gaoikawa,
jordan-turbofish,
kurtmcmillan,
riley-dixon,
sbates130272 and
zbyrne
as code owners
October 14, 2026 10:44
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Critical connection establishment, resource lifetime, parsing, security, and benchmark-validity issues remain unresolved.
29 open findings
And 9 more that still need to be addressed.
What changed in this PR
Adds an RC-RDMA v3 transport using RDMA CM, CPU-only RDMA support, and an AWS SDK integration demo.
Changes:
- Adds v3 wire encoding and a passive RC-CM listener.
- Adds CPU-only RDMA and updates existing integrations/tests.
- Adds an AWS S3 client and benchmarking demo.
| File | Description |
|---|---|
test/unit/test-v3-wire.cpp |
Tests v3 token/reply encoding. |
test/unit/test-transport.cpp |
Adapts transport tests for v1/v3 and CPU mode. |
test/unit/test-thread-safety.cpp |
Adds v3 reply handling. |
test/unit/test-state.cpp |
Adjusts state tests by wire mode. |
test/unit/test-staging.cpp |
Supports the HIP shim. |
test/unit/test-seams.cpp |
Adapts seams and replies for CPU/v3. |
test/unit/test-rc-listener.cpp |
Adds listener helper tests. |
test/unit/test-ibv-lazy-load.cpp |
Disables lazy-load coverage. |
test/unit/test-api-exceptions.cpp |
Makes exception tests wire-aware. |
test/unit/test-api-args.cpp |
Validates selected token length. |
test/unit/fake-device.h |
Uses the CPU HIP shim. |
test/unit/fake-device.cpp |
Supplies default v3 replies. |
test/unit/CMakeLists.txt |
Builds v3/listener tests and dependencies. |
test/integration/rdma-test-server/CMakeLists.txt |
Supports CPU-only builds. |
test/integration/CMakeLists.txt |
Adds the AWS integration. |
test/integration/aws-s3-client/log.h |
Adds integration logging. |
test/integration/aws-s3-client/hipobj_aws_s3_client.h |
Declares the AWS RDMA client. |
test/integration/aws-s3-client/hipobj_aws_s3_client.cpp |
Implements AWS control requests. |
test/integration/aws-s3-client/CMakeLists.txt |
Builds the AWS client/demo. |
test/integration/aws-s3-client/aws_s3_demo.cpp |
Adds benchmarking and verification. |
src/state.h |
Stores listener state and wire selection. |
src/state.cpp |
Initializes and tears down the listener. |
src/s3/control.h |
Declares v3 reply reception. |
src/s3/control.cpp |
Decodes v3 control replies. |
src/rdma/everpure/v3-wire.h |
Defines the v3 wire protocol. |
src/rdma/everpure/v3-wire.cpp |
Implements v3 serialization. |
src/rdma/everpure/rc-listener.h |
Declares the RC-CM listener. |
src/rdma/everpure/rc-listener.cpp |
Implements connection handling. |
src/hipobj.cpp |
Integrates v3 transfers and public parsing. |
src/common/rdma-topology.cpp |
Removes a direct HIP include. |
src/common/pd-lock.h |
Declares PD lifecycle serialization. |
src/common/pd-lock.cpp |
Implements the PD mutex guard. |
src/common/log.h |
Adds internal logging. |
src/common/hip-shim.h |
Provides CPU-only HIP substitutes. |
src/common/hip-seam.h |
Selects HIP runtime or shim. |
src/common/hip-seam.cpp |
Disables DMA-BUF operations in CPU mode. |
src/CMakeLists.txt |
Builds and links v3 components. |
src/buffer.cpp |
Serializes MR lifecycle operations. |
integrations/minio-cpp/CMakeLists.txt |
Supports CPU-only linking. |
include/hipobj.h |
Documents v3 and adds its reply parser. |
examples/put-object.cpp |
Supports the HIP shim. |
examples/get-object.cpp |
Supports the HIP shim. |
examples/CMakeLists.txt |
Configures CPU-only examples. |
docs/rc-rdma-flow.md |
Documents v3 transfer flows. |
CMakeLists.txt |
Adds CPU, AWS, and wire-mode options. |
cmake/HIPOBJUseGTest.cmake |
Propagates new build definitions. |
cmake/HIPOBJFetchAwsSdkCpp.cmake |
Finds or fetches AWS SDK C++. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…egration Defines hipobj-rc-v3, the wire format and client-side implementation for RDMA-accelerated S3 PUT/GET against an S3 server speaking RC RDMA. hipObject runs a passive RDMA CM listener (src/rdma/everpure/rc-listener.*) and advertises its address plus a registered buffer's rkey/remoteAddr/length to the S3 server via the x-amz-rc-rdma-token HTTP header (src/rdma/everpure/v3-wire.*). The S3 server dials back into that listener to establish the RC connection, then performs the actual transfer as a one-sided RDMA READ (PUT) or WRITE (GET) directly against hipObject's registered memory, reporting completion via the x-amz-rc-rdma-reply header. RDMA CM negotiates the QP-level connection setup itself, so no QP number/GID/LID exchange is needed on the wire, unlike the older v1 scheme. Also includes test/integration/aws-s3-client, an AWS-SDK-based S3 client/demo used to exercise and benchmark the RC RDMA path end-to-end (PUT/GET timing, checksum verification, parallel-transfer testing) against a real S3 server. The demo signs requests only when AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set in the environment; it doesn't read ~/.aws/credentials. Without them it sends unsigned requests: that's for demo use against a simple s3-server, which doesn't check auth. An S3 server that checks auth needs both variables set.
- Listener: binds only to a specific address (HIPOBJ_RC_LISTEN_IP or the NIC's IPv4 GID) and allocates its PD on librdmacm's context, so the default path accepts connections. Buffers register on its PD. - Init: a v3 hipObjInit() fails when the listener can't start, and no token is built without one. - Listener interface: RcListener is implemented by RdmaCmRcListener and, in tests, by a fake. startRcListener() starts it, and Instance owns it through RcListenerPtr, which stops it. - Teardown: buffers are deregistered before the listener frees its PD. - Async events: one event per epoll wake-up, so the loop can't block. - Wire: v3 tokens and replies are parsed strictly with parseNumber(), using std::string_view and std::optional as v1's parsers do. - Transfers: a v3 GET/PUT fails unless the reply reports the full size. - Logs: no token, rkey or remote address. - Tests: fakes report the token's length, and a fake listener lets v3 init run without RDMA hardware. New tests for parsing, the byte count, init failures, the advertised address and deregisterAll(). - TODO in HandleConnectRequest(): cap and age out connections.
- Add the HIPOBJ_RDMA_WIRE_V3 CMake option, OFF by default, in place of HIPOBJ_RDMA_WIRE_V1. A default build speaks v1, and neither compiles SystemRdmaCm nor links librdmacm. - HIPOBJ_AWS_S3_CLIENT requires v3, and HIPOBJ_MINIO_CLIENT requires v1. - Call librdmacm through an RdmaCm interface on the Platform (rdmacm-core.h, rdmacm-wrapper.*), and add the async event calls to Verbs. The listener uses the RdmaCm and Verbs it's started with. - Remove the RcListener interface, its test hook and the static listener registry. hipObjInit() starts the listener before building the Instance, which owns it and stops it after deregistering the buffers. - Read HIPOBJ_RC_LISTEN_IP and HIPOBJ_RC_LISTEN_PORT through Options. v3 creates no QPs of its own. - Remove PdLockGuard: libibverbs objects are thread safe. The listener's lock guards only its connection list and count. - Replace the printf-style log macros, in the library and the AWS client, with std::format functions that print with printTo() and never throw. Debug messages with costly arguments check logEnabled() first. - Encode v3 tokens and replies into spans, without memcpy, snprintf or C arrays. - Fix the GCC, Clang and clang-tidy findings, and format with clang-format 18. - hipobj.h names each wire's headers, and hipObjTokenClientNic() reads the token of the wire that's built. - AWS client: percent-encode object keys, refuse a PUT with a body, register offset + size bytes, and don't call hipObjShutdown() during static destruction. The SDK fetch restores BUILD_SHARED_LIBS. - AWS demo: check the parallel transfer size for overflow, and with real HIP, fill and checksum the GPU buffer through host memory. - CPU mode: include the HIP header in abandoned-copies.cpp only with ROCm, define hipStreamPerThread in hip-shim.h, and link Threads::Threads. - Tests: FakeRdmaCm drives the real listener; the pool tests are v1-only; new tests keep line coverage above 90%. - Docs: rc-rdma-flow.rst in the user guide, the new options and environment variables in building.rst, and CHANGELOG entries for them and hipObjParseRdmaReplyV3().
surya-everpure
force-pushed
the
everpure-rc-rdma
branch
from
October 22, 2026 18:19
0ab76c3 to
7610981
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Defines hipobj-rc-v3, the wire format and client-side implementation for RDMA-accelerated S3 PUT/GET against an S3 server speaking RC RDMA.
hipObject runs a passive RDMA CM listener (src/rdma/everpure/rc-listener.) and advertises its address plus a registered buffer's rkey/remoteAddr/length to the S3 server via the x-amz-rc-rdma-token HTTP header (src/rdma/everpure/v3-wire.). The S3 server dials back into that listener to establish the RC connection, then performs the actual transfer as a one-sided RDMA READ (PUT) or WRITE (GET) directly against hipObject's registered memory, reporting completion via the x-amz-rc-rdma-reply header. RDMA CM negotiates the QP-level connection setup itself, so no QP number/GID/LID exchange is needed on the wire, unlike the older v1 scheme.
Also includes test/integration/aws-s3-client, an AWS-SDK-based S3 client/demo used to exercise and benchmark the RC RDMA path end-to-end (PUT/GET timing, checksum verification, parallel-transfer testing) against a real S3 server.
The demo signs requests only when AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set in the environment; it doesn't read ~/.aws/credentials. Without them it sends unsigned requests: that's for demo use against a simple s3-server, which doesn't check auth. An S3 server that checks auth needs both variables set.
Submission Checklist