(more) precisely track memory accesses and allocations across FFI #4326

nia-e · 2025-05-17T22:52:50Z

This greatly expands the information we have access to when code is executed across the FFI bound, including tracking exact allocations/deallocations (where possible) and accesses to Miri memory exposed to foreign code.

The current architecture forks Miri into 2 processes at the point of the first FFI call, with the parent ptraceing the child. Communication is done via IPC channels; the child process signals when to start or stop tracing (during FFI), and the parent returns information about memory accesses that occurred, if any.

Alternatives considered:

FUSE: It would be possible to obtain most of the same information about accesses to our memory (but not allocations done on C's side) by registering a FUSE fs which lives entirely in memory and mmaping files from there to act as the machine's memory. The main benefits of this approach would be that it would be more portable as we'd no longer need to disassemble the foreign code to examine it (though we will still need to read arbitrary registers on the fly) and that it would give us slightly more accurate information about the sizes and types of memory accesses where the current approach lightly conservatively overestimates. However, I decided not to do this because it requires FUSE utils to actually be installed on the host system and would add a lot more complexity to the logic
Plain sigactions: We can also get data by registering sigactions to trigger when foreign code accesses our memory, but even though this is much simpler it would only give us page-grained information about what was performed (where the current approach is more or less byte-grained)

As it stands, this PR is not ready for merging pending at least the following being resolved. Still, I am opening this because significant sections of the code are ready for (preliminary, this is still very WIP) review. In ~decreasing order of severity as I see it:

Actually incorporate the obtained information into Miri's memory tracking instead of just printing to stderr
General cleanup such as removing hardcoded unwraps, adding more documentation, etc
Dynamically fall back to the old behaviour without panicking if the parent fails to trace properly, e.g. if ptrace is disabled
Add more tests
Add logic for non-x86 architectures
Handle the foreign code spawning multiple threads

Open questions / fixable current limitations:

Do we want to pull in an existing allocator instead of having our own for managing the machine memory? (No; we merged our own)
Can this be made to be combineable with a multi-seeded Miri run? (Yes!)
What happens if the FFI code forks further (do we even allow it)?
Should we switch out iced_x86 with something else (e.g. go ahead with the FUSE idea, or use Capstone/llvm-sys for the better compatibility and eat the binary size & compile time increase)? (Switched to Capstone)

Zulip thread: https://rust-lang.zulipchat.com/#narrow/channel/269128-miri/topic/Initializing.20Miri.20memory.20from.20C.20FFI/with/518542321

Works toward (I hesitate to say closes) #11

RalfJung · 2025-05-21T09:54:01Z

Write-up of design options and choices: https://hackmd.io/@scZ140SZTaOb1VASX_ju-Q/SJUzIeo-eg

nia-e · 2025-05-21T10:56:41Z

This doesn't build or pass tests right now since it builds against my fork of rustc, but against it it passes all tests on x86 linux and (assuming I didn't mess them up) on other platforms as well!

nia-e · 2025-05-21T10:57:16Z

@rustbot ready

RalfJung · 2025-05-22T11:57:52Z

@rustbot author
based on discussion on Zulip re: splitting up the PR

rustbot · 2025-05-22T11:57:56Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

nia-e · 2025-05-31T07:11:55Z

I think this is ready for at least a preliminary review? The rustc side I'm pretty settled on, though, so I'll push and mark that as ready also. There's every chance I missed something really obvious and a second pair of eyes probably helps.

@rustbot ready

rustbot · 2025-06-06T10:18:43Z

☔ The latest upstream changes (possibly 8344967) made this pull request unmergeable. Please resolve the merge conflicts.

RalfJung

I started idly scrolling through this, but then realized this is way too much for me at the moment.^^ It is unlikely I'll be able to review a PR of this size and complexity before mid-July. I expect I have to book at least half a day for this, and I don't have that much time to spare earlier than that. If there's some way we could arrange real-time communication (e.g. via zoom) for a quicker feedback cycle, that could be useful.

Is there any way to split this up? Like

first land some basic ptrace setup that does basically nothing
then add catching of memory events in the server (but just logging the info to stderr or so without communicating them to Miri proper)
then add a communication channel so that core Miri learns about these events and uses them for something useful

Also, the IPC protocol between server and client should be explicitly documented somewhere: what messages are there, what is their format, what do they mean, when can which message be sent by whom and what is the reply. Since it seems there are three channels, such docs are needed for each of them. Probably the file that defines the message types it the best place for that. Currently that's just the crate root of trace I think; this should be a proper module on its own.

RalfJung · 2025-06-11T19:14:57Z

src/bin/miri.rs

+        #[cfg(target_os = "linux")]
+        if !config.native_lib.is_empty() && !config.force_old_native_lib {
+            // FIXME: This should display a diagnostic / warning on error
+            // SAFETY: No other threads have spawned yet


This comment is almost certainly wrong. rustc can spawn an arbitrary amount of threads before calling us here.

The forking should happen way earlier, probably right after argument parsing but before we invoke rustc_driver.

RalfJung · 2025-06-11T19:22:14Z

no_alloc.json

This file probably should not be here. ;)

nia-e · 2025-06-15T13:29:20Z

I'd be more than happy to do that, yeah! I've also been quite busy the last couple of weeks and so haven't had much time to address reviews, but I'm okay with talking directly and splitting this up further. Let me know what and when works for you, I've had a bit of trouble with Zoom on my device in the past so I have a slight preference for Jitsi but it's not a huge deal. I'm likely to be free again and address review comments + be able to talk sometime early next week

nia-e · 2025-06-16T14:41:39Z

@rustbot author
based on discussion above re: splitting this up further

RalfJung · 2025-06-19T02:04:27Z

Okay, sounds good! Let's see if it will be needed with the split-up approach and Oli helping with reviews.

fix review comments ?? docs move init to arg parsing docs again

nia-e force-pushed the ffi-ptrace branch from 3d31e12 to 7f085ad Compare May 18, 2025 00:53

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-author Status: Waiting for the PR author to address review comments label May 19, 2025

nia-e force-pushed the ffi-ptrace branch 2 times, most recently from 6df5ade to 45232b8 Compare May 19, 2025 21:03

nia-e force-pushed the ffi-ptrace branch from 1f00150 to 02f6d50 Compare May 21, 2025 10:52

rustbot added S-waiting-on-review Status: Waiting for a review to complete and removed S-waiting-on-author Status: Waiting for the PR author to address review comments labels May 21, 2025

nia-e marked this pull request as ready for review May 21, 2025 11:03

nia-e changed the title ~~WIP: (more) precisely track memory accesses and allocations across FFI~~ (more) precisely track memory accesses and allocations across FFI May 21, 2025

nia-e force-pushed the ffi-ptrace branch 5 times, most recently from b2cb76e to 85189c3 Compare May 22, 2025 11:56

rustbot added S-waiting-on-author Status: Waiting for the PR author to address review comments and removed S-waiting-on-review Status: Waiting for a review to complete labels May 22, 2025

This was referenced May 22, 2025

Isolate MiriMachine memory from Miri's #4343

Merged

rustc_const_eval: Expose APIs for signalling foreign accesses to memory rust-lang/rust#141391

Open

nia-e force-pushed the ffi-ptrace branch 6 times, most recently from b856447 to ae13c9a Compare May 23, 2025 15:56

nia-e force-pushed the ffi-ptrace branch 3 times, most recently from 6ac4531 to 3d6ca8f Compare May 31, 2025 06:42

rustbot added S-waiting-on-review Status: Waiting for a review to complete and removed S-waiting-on-author Status: Waiting for the PR author to address review comments labels May 31, 2025

nia-e force-pushed the ffi-ptrace branch 3 times, most recently from 0ea7416 to f97ebcf Compare June 3, 2025 21:13

This comment has been minimized.

Sign in to view

nia-e force-pushed the ffi-ptrace branch from 84efa6e to 7d1ce18 Compare June 4, 2025 16:18

RalfJung reviewed Jun 11, 2025

View reviewed changes

nia-e force-pushed the ffi-ptrace branch 2 times, most recently from 88170db to ef5b4c7 Compare June 15, 2025 19:58

nia-e mentioned this pull request Jun 15, 2025

trace: add barebones ptrace setup #4401

Merged

rustbot added S-waiting-on-author Status: Waiting for the PR author to address review comments and removed S-waiting-on-review Status: Waiting for a review to complete labels Jun 16, 2025

nia-e force-pushed the ffi-ptrace branch from ffd5c29 to a442f32 Compare June 18, 2025 14:25

nia-e mentioned this pull request Jun 18, 2025

trace: implement supervisor components for tracing #4405

Merged

nia-e added 6 commits June 19, 2025 15:04

ffi ptracing

a6b031d

fix review comments ?? docs move init to arg parsing docs again

fix the early init

c5c2967

nit

18c0618

nits

0c1e14b

rebase

083d635

zero out stack, etc

e1d4633

nia-e force-pushed the ffi-ptrace branch from a442f32 to e1d4633 Compare June 19, 2025 13:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

(more) precisely track memory accesses and allocations across FFI #4326

(more) precisely track memory accesses and allocations across FFI #4326

Uh oh!

nia-e commented May 17, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

RalfJung commented May 21, 2025

Uh oh!

nia-e commented May 21, 2025

Uh oh!

nia-e commented May 21, 2025

Uh oh!

RalfJung commented May 22, 2025

Uh oh!

rustbot commented May 22, 2025

Uh oh!

nia-e commented May 31, 2025

Uh oh!

This comment has been minimized.

rustbot commented Jun 6, 2025

Uh oh!

RalfJung left a comment •

edited

Loading

Uh oh!

RalfJung Jun 11, 2025 •

edited

Loading

Uh oh!

RalfJung Jun 11, 2025

Uh oh!

nia-e Jun 15, 2025

Uh oh!

nia-e commented Jun 15, 2025

Uh oh!

nia-e commented Jun 16, 2025

Uh oh!

RalfJung commented Jun 19, 2025

Uh oh!

Uh oh!

(more) precisely track memory accesses and allocations across FFI #4326

Are you sure you want to change the base?

(more) precisely track memory accesses and allocations across FFI #4326

Uh oh!

Conversation

nia-e commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

RalfJung commented May 21, 2025

Uh oh!

nia-e commented May 21, 2025

Uh oh!

nia-e commented May 21, 2025

Uh oh!

RalfJung commented May 22, 2025

Uh oh!

rustbot commented May 22, 2025

Uh oh!

nia-e commented May 31, 2025

Uh oh!

This comment has been minimized.

rustbot commented Jun 6, 2025

Uh oh!

RalfJung left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RalfJung Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RalfJung Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

nia-e Jun 15, 2025

Choose a reason for hiding this comment

Uh oh!

nia-e commented Jun 15, 2025

Uh oh!

nia-e commented Jun 16, 2025

Uh oh!

RalfJung commented Jun 19, 2025

Uh oh!

Uh oh!

nia-e commented May 17, 2025 •

edited

Loading

RalfJung left a comment •

edited

Loading

RalfJung Jun 11, 2025 •

edited

Loading