diff --git a/projects/hrr/PRIVACY.md b/projects/hrr/PRIVACY.md new file mode 100644 index 000000000000..7c9af5dbf2e5 --- /dev/null +++ b/projects/hrr/PRIVACY.md @@ -0,0 +1,109 @@ +# What an HRR archive contains, and what replaying one means + +This page states what a capture archive can and cannot contain, and what a person who +replays one is trusting. It is the customer-facing text of +[AIRUNTIME-2733](https://amd-hub.atlassian.net/browse/AIRUNTIME-2733) (capture privacy and +content control). The engineering threat model is +[AIRUNTIME-2853](https://amd-hub.atlassian.net/browse/AIRUNTIME-2853). + +Every task under AIRUNTIME-2733 that changes what an archive holds updates this page in the +same change. `tools/tests/test_privacy_doc.py`, run by the ctest `hrr-generator-rules`, +fails when the manifest fields written by capture and the table below disagree. + +## Replay runs GPU code taken from the archive + +An archive is untrusted executable input for the tool that replays it. `hrr-playback` loads +the code objects stored in the archive and launches their kernels on the GPU. That code runs +with the replaying process's access to device memory and to any host memory mapped for the +GPU, and nothing in the replay tool can check what it does. The tool refuses malformed +records and caps memory, events and run time, but it cannot make the code itself safe. +Replay maps recorded pointers to the buffers it allocated so that the run can be rebuilt; +this is reconstruction, not a safety boundary. Only a value that matches a recorded +allocation is translated. A pointer argument that matches none is passed as null if it could +be a device address. A pointer argument that cannot be an address, such as a small sentinel, +is passed as recorded. Every other argument byte is passed as recorded, including a +pointer-sized word inside a struct that matches none. A kernel can therefore still receive +addresses from the original process. + +Replay an archive only if you trust whoever produced it, as an unprivileged user, on a +machine and GPU that hold no data you would not hand to that person. A container is a +reasonable boundary. `hrr-playback --info` reads the archive without loading any code and +needs no GPU. + +## Capture modes + +Capture is disabled by default on AMD platforms at present (see the [README](README.md)); +where it is built in, it is armed by `HIP_HRR_CAPTURE_OUTPUT` alone. Today there is one +mode, and it records **contents**: everything in the tables below. + +[AIRUNTIME-2880](https://amd-hub.atlassian.net/browse/AIRUNTIME-2880) adds a +**metadata-only** mode as the default and moves contents behind a second variable. The +"Mode" column states which mode records each item once that task has landed. Until then, +every capture is a contents capture. + +## What an archive contains + +| Item | Where | Mode | Removed or reduced by | +|---|---|---|---| +| Sequence of successful HIP calls, with sizes, flags, device addresses, handles, thread ids | `events.bin` | both | — kept, replay needs it | +| Kernel names, mangled but read back with `c++filt`; they describe the model's architecture and precision | `events.bin` | both | — kept, triage names the failing kernel | +| Event timestamps, from a monotonic clock (not the date or time of day) | `events.bin` | both | — kept | +| Device properties passed to `hipChooseDevice`, the whole `hipDeviceProp_t`: device and architecture names, PCI ids and, in the current layout, the UUID and LUID | `events.bin` | both | — kept, the call's answer depends on them | +| Other structures passed into a HIP call, copied whole, such as copy, memset, graph node, allocation, pool and launch-attribute parameters. Padding, reserved and unused union bytes are copied as the caller left them | `events.bin` | both | — kept, replay needs it | +| Host buffers copied to or from the device, byte for byte, unencrypted. For a serving stack this includes prompts and generated output as token ids, which the model's public tokeniser reverses, and the model weights | `blobs/` | contents | [AIRUNTIME-2880](https://amd-hub.atlassian.net/browse/AIRUNTIME-2880) | +| Host memory registered with `hipHostRegister`: the whole range, byte for byte, as it was at registration, whether or not it is ever copied | `blobs/` | contents | [AIRUNTIME-2880](https://amd-hub.atlassian.net/browse/AIRUNTIME-2880) | +| Kernel argument bytes | `events.bin` | contents | [AIRUNTIME-2880](https://amd-hub.atlassian.net/browse/AIRUNTIME-2880) | +| Code objects the process loaded (the application's own GPU binary code) | `code_objects/`, `blobs/` | contents | [AIRUNTIME-2880](https://amd-hub.atlassian.net/browse/AIRUNTIME-2880) | +| Linker input buffers passed to `hipLinkAddData`, every byte, with their names, including inputs the call rejected and inputs never linked or loaded | `blobs/` | contents | [AIRUNTIME-2880](https://amd-hub.atlassian.net/browse/AIRUNTIME-2880) | +| Whole fat binary bundles, including targets HIP never loads | `blobs/` | contents | [AIRUNTIME-2858](https://amd-hub.atlassian.net/browse/AIRUNTIME-2858): only the code objects HIP loads | +| Debug and comment sections of code objects: absolute build paths, source file names, the exact compiler build | `code_objects/`, `blobs/` | contents | [AIRUNTIME-2881](https://amd-hub.atlassian.net/browse/AIRUNTIME-2881): stripped | +| Framework allocator layout, when a region producer ran beside capture | `regions/` | both | — kept, written only by an opt-in producer | +| Marker that capture is running: on Linux the boot id and the process start time in clock ticks since boot; on Windows the process creation date and time. A normal exit removes it; a process that crashes or is killed leaves it | `active` | both | — kept, region producers read it | + +The per-process `manifest.json` also holds `pid`, `parent_pid`, `complete`, `event_count`, +`blob_count`, the HIP and comgr versions, and one block per device: + +| Device field | Kept | Removed by | +|---|---|---| +| `name`, `gcn_arch_name`, `compute_capability` | yes | — | +| `total_global_mem`, `multi_processor_count`, `warp_size`, `max_threads_per_block` | yes | — | +| `clock_rate_khz`, `memory_clock_rate_khz`, `memory_bus_width`, `l2_cache_size` | yes | — | +| `integrated`, `managed_memory`, `memory_pools_supported`, `compute_mode` | yes | — | +| `uuid`, `pci`, `luid`, `luid_device_node_mask`: identify the individual GPU and its slot | no | [AIRUNTIME-2881](https://amd-hub.atlassian.net/browse/AIRUNTIME-2881): blanked | + +This table covers `manifest.json` only. A `hipChooseDevice` call can carry the same +identifiers in `events.bin`, as listed above. + +## Never recorded + +Capture does not read the host name, the command line, environment variables or the user +name, and records a date and time of day only in the `active` marker on Windows. A string +reaches the archive only as a kernel name; as a string argument of a HIP call (the path +given to `hipLinkAddFile`, the name given to `hipLinkAddData`, the function name given to +`hipGetProcAddress`); as the name of a device variable the program registers; as a character +array in a structure passed to a HIP call (the device and architecture names given to +`hipChooseDevice`); inside the padding or unused bytes of such a structure; inside a host +buffer the workload copied or registered; inside a linker input buffer, even one the linker +rejected; or inside a code object. A path in such a buffer, +for example a compiler cache path under the user's home directory, is recorded with it. +Memory that never crossed a HIP call is not recorded, and neither is a failed call, apart +from the exceptions listed in [skills/enable-recording/SKILL.md](skills/enable-recording/SKILL.md). + +## What no mode can promise + +A metadata-only archive with identifiers blanked still names every kernel the workload +ran, which describes the model and the serving stack, and still records every size and +address. It holds no buffer contents and no code, so it cannot reproduce a fault that +depends on values, and it cannot be replayed in full. A contents archive is as sensitive +as the data the workload handled: where it may go has to be agreed before it is sent. + +## How the archive is protected on disk + +On Linux the archive is readable only by the user who captured it (directories 0700, files +0600). [AIRUNTIME-2847](https://amd-hub.atlassian.net/browse/AIRUNTIME-2847) makes every +write go through the directory descriptors capture checked. +[AIRUNTIME-2857](https://amd-hub.atlassian.net/browse/AIRUNTIME-2857) decides whether +capture must ask for consent before it starts. +[AIRUNTIME-2878](https://amd-hub.atlassian.net/browse/AIRUNTIME-2878) keeps the crash-time +flush without displacing the application's own crash handler. None of these changes what an +archive contains. diff --git a/projects/hrr/README.md b/projects/hrr/README.md index 02bad2355acf..1bffe994b781 100644 --- a/projects/hrr/README.md +++ b/projects/hrr/README.md @@ -93,7 +93,7 @@ User-facing capture, replay, and validation knobs. Implementation details can be | `HIP_HRR_CAPTURE_OUTPUT` | *(unset)* | Enable capture; path to the `.hrr` archive directory. An empty or blank value leaves capture off. Ignored, with one line on stderr, when Linux starts the program in secure-execution mode (set-user-ID, set-group-ID, file capabilities or an LSM transition) | | `HIP_HRR_DEBUG_ARGS` | off | Dump every captured kernel arg to the log (debug / provenance) | -On Linux the archive is readable only by the user who captured it (directories 0700, files 0600). If the archive cannot be set up, capture is disabled with a `[HRR capture] Capture disabled` message on stderr and the application runs normally. +On Linux the archive is readable only by the user who captured it (directories 0700, files 0600). What an archive contains, which of it each capture mode records, and why replaying an archive means running its GPU code: [PRIVACY.md](PRIVACY.md). If the archive cannot be set up, capture is disabled with a `[HRR capture] Capture disabled` message on stderr and the application runs normally. ### `hrr-playback` CLI options diff --git a/projects/hrr/skills/enable-recording/SKILL.md b/projects/hrr/skills/enable-recording/SKILL.md index 33721c99a87d..228c070c860c 100644 --- a/projects/hrr/skills/enable-recording/SKILL.md +++ b/projects/hrr/skills/enable-recording/SKILL.md @@ -134,7 +134,7 @@ What the archive holds, per process, and it is worth reading out to whoever has - **`manifest.json`**: process and parent process ids, event and blob counts, the clean-shutdown flag, the HIP and comgr versions that captured, and the device's name, architecture, PCI location and UUID. - **`regions/*.hrrr`**: the framework allocator's own layout, when a producer ran alongside capture. -Apart from the few exceptions in step 2, failed HIP calls are not recorded, and nothing that never crossed a HIP call is. The archive is exactly as sensitive as the data the workload handled, so where it may go has to be agreed before it is uploaded. +Apart from the few exceptions in step 2, failed HIP calls are not recorded, and nothing that never crossed a HIP call is. The archive is exactly as sensitive as the data the workload handled, so where it may go has to be agreed before it is uploaded. [PRIVACY.md](../../PRIVACY.md) is the page to hand to whoever approves it. `verify --json` gives each process's counts and sizes in a form you can attach to the request; the device details are in each `manifest.json`. For a multi-process archive, `hrr-playback --info` against the archive root prints a table covering every process at once. The cross-check in `verify` reports on one process, which is enough to judge the capture and not enough for a disclosure inventory. diff --git a/projects/hrr/tests/CMakeLists.txt b/projects/hrr/tests/CMakeLists.txt index d7843e1d645f..88009139715f 100644 --- a/projects/hrr/tests/CMakeLists.txt +++ b/projects/hrr/tests/CMakeLists.txt @@ -43,7 +43,9 @@ add_subdirectory(unit) # Generator rules: regenerate the playback handlers and fail if # one still casts an archive field to a pointer or an opaque handle. # Also fails if a checked-in generated source or the API matrix header is -# stale against the generator (test_generated_files_current.py). +# stale against the generator (test_generated_files_current.py), and if +# PRIVACY.md no longer lists the manifest fields capture writes +# (test_privacy_doc.py). find_package(Python3 COMPONENTS Interpreter) if(Python3_Interpreter_FOUND) add_test(NAME hrr-generator-rules diff --git a/projects/hrr/tools/tests/test_privacy_doc.py b/projects/hrr/tools/tests/test_privacy_doc.py new file mode 100644 index 000000000000..8d49326c05f6 --- /dev/null +++ b/projects/hrr/tools/tests/test_privacy_doc.py @@ -0,0 +1,131 @@ +# Copyright (c) Advanced Micro Devices, Inc., or its affiliates. +# +# SPDX-License-Identifier: MIT +"""PRIVACY.md lists what capture writes into an archive. + +The page is the customer text for AIRUNTIME-2733. Its device table has to name +exactly the fields hip_capture_metadata.cpp writes into a manifest, so a task +that adds, removes or blanks one cannot land without updating the page. Its +inventory has to list the active marker, the hipHostRegister snapshot, the +hipLinkAddData inputs and the hipDeviceProp_t snapshots in events.bin. + +Run: python3 -m unittest discover -s projects/hrr/tools/tests +""" + +import re +import unittest +from pathlib import Path + +HRR = Path(__file__).resolve().parents[2] +PRIVACY = HRR / "PRIVACY.md" +CAPTURE_SRC = HRR.parent / "clr" / "hipamd" / "src" / "hrr" +METADATA_CPP = CAPTURE_SRC / "hip_capture_metadata.cpp" + +# The tasks under AIRUNTIME-2733 that the page links, one per statement. +EPIC_TASKS = ( + "AIRUNTIME-2847", + "AIRUNTIME-2857", + "AIRUNTIME-2858", + "AIRUNTIME-2878", + "AIRUNTIME-2880", + "AIRUNTIME-2881", +) + + +def device_fields_written() -> set: + """The keys append_prop_fields writes into each manifest device block.""" + text = METADATA_CPP.read_text(encoding="utf-8") + body = text.split("void append_prop_fields(", 1)[1].split("\n}\n", 1)[0] + return set(re.findall(r'\\"(\w+)\\": ', body)) - {"properties"} + + +def inventory_rows() -> list: + """The rows of the page's "What an archive contains" table.""" + text = PRIVACY.read_text(encoding="utf-8") + table = text.split("| Item | Where |", 1)[1].split("\n\n", 1)[0] + return table.splitlines()[2:] + + +def device_fields_documented() -> set: + """The backticked names in the first column of the page's device table.""" + text = PRIVACY.read_text(encoding="utf-8") + table = text.split("| Device field |", 1)[1].split("\n\n", 1)[0] + fields = set() + for row in table.splitlines()[2:]: + fields.update(re.findall(r"`(\w+)`", row.split("|")[1])) + return fields + + +class PrivacyDocTest(unittest.TestCase): + def test_device_table_matches_the_manifest_capture_writes(self): + written = device_fields_written() + self.assertIn("uuid", written) + self.assertEqual(device_fields_documented(), written) + + def test_inventory_lists_the_active_marker_capture_writes(self): + writer = (CAPTURE_SRC / "hip_capture_writer.cpp").read_text(encoding="utf-8") + name = re.search(r'kActiveMarker = "/(\w+)"', writer).group(1) + self.assertTrue( + any(row.split("|")[2].strip() == f"`{name}`" for row in inventory_rows()) + ) + + def assert_blob_row(self, api: str) -> tuple: + """api's capture wrapper writes a blob and the inventory has its row.""" + capture = (CAPTURE_SRC / "hip_capture.cpp").read_text(encoding="utf-8") + body = capture.split(f"hipError_t capture_{api}(", 1)[1] + body = body.split("\n}\n", 1)[0] + self.assertIn("write_blob(", body) + rows = [r for r in inventory_rows() if f"`{api}`" in r and "`blobs/`" in r] + self.assertTrue(rows, api) + return body, rows[0] + + def test_inventory_lists_the_host_register_snapshot(self): + self.assert_blob_row("hipHostRegister") + + def test_inventory_lists_linker_inputs_rejected_ones_included(self): + body, row = self.assert_blob_row("hipLinkAddData") + # The row says rejected inputs are recorded: true while the wrapper + # does not gate on the call's result. + self.assertNotIn("hipSuccess", body) + self.assertIn("rejected", row) + + def test_lists_device_property_snapshots_outside_the_manifest(self): + # Every shim that copies a caller's hipDeviceProp_t into its event + # records the device name, PCI ids, UUID and LUID in events.bin. + shims = (CAPTURE_SRC / "hip_capture_generated.cpp").read_text( + encoding="utf-8" + ) + (CAPTURE_SRC / "hip_capture.cpp").read_text(encoding="utf-8") + apis = set() + for shim in re.split(r"\n(?=(?:static )?\S+\*? capture_)", shims): + name = re.match(r"(?:static )?\S+\*? capture_(\w+)\(", shim) + if name and re.search(r"memcpy\(a\.\w+, \w+, sizeof\(hipDeviceProp", shim): + apis.add(re.sub(r"R\d{4}$", "", name.group(1))) + self.assertIn("hipChooseDevice", apis) + text = PRIVACY.read_text(encoding="utf-8") + strings = text.split("## Never recorded", 1)[1].split("\n## ", 1)[0] + for api in apis: + rows = [r for r in inventory_rows() if f"`{api}`" in r] + self.assertTrue(any("`events.bin`" in r and "UUID" in r for r in rows), api) + self.assertIn(f"`{api}`", strings) + + def test_states_that_an_archive_is_untrusted_executable_input(self): + text = PRIVACY.read_text(encoding="utf-8") + self.assertIn("untrusted executable input", text) + self.assertIn("launches their kernels on the GPU", text) + + def test_links_each_epic_task(self): + text = PRIVACY.read_text(encoding="utf-8") + for key in EPIC_TASKS: + self.assertIn(f"https://amd-hub.atlassian.net/browse/{key}", text) + + def test_readme_and_recording_skill_link_the_page(self): + readme = (HRR / "README.md").read_text(encoding="utf-8") + skill = (HRR / "skills" / "enable-recording" / "SKILL.md").read_text( + encoding="utf-8" + ) + self.assertIn("](PRIVACY.md)", readme) + self.assertIn("](../../PRIVACY.md)", skill) + + +if __name__ == "__main__": + unittest.main()