Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ That uses GitHub Actions cache by default. For S3-backed caching shared across r

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Docs say unrecognized names “such as custom scripts, ccache, or cargo-zigbuild shims” are detected via -E probing. Bare ccache is still in recognizes_rejects_non_c_compilers as not a compiler, and a real ccache argv0 will fail the probe (ccache expects ccache <compiler> …). cargo-zigbuild’s zigcc-* names were already on the allowlist. The docs oversell what the probe does for nested wrappers (kache ccache gcc still has argv0 ccache).

Suggestion: Rephrase to “custom compiler wrappers whose executable itself accepts the usual preprocessor CLI (forwards -E and defines __clang__/__GNUC__)”, and give a concrete example (symlink/copy of gcc, or a thin shim). Mention that nested multi-token wrappers are not expanded.

## C/C++ caching

Alongside the rustc wrapper, kache can cache **C/C++ object compiles** as a `cc` / `c++` wrapper. It recognizes `cc`, `c++`, `gcc`, `g++`, `clang`, and `clang++` (plus versioned variants like `gcc-13` and target-prefixed cross compilers like `arm-linux-gnueabihf-gcc`), and `clang-cl` or any `--driver-mode=cl` invocation (clang in MSVC driver mode) — on every OS, not just Windows, since recognition keys off the driver mode rather than the host. clang-cl is what mozconfigs and the `cc` crate use on Windows:
Alongside the rustc wrapper, kache can cache **C/C++ object compiles** as a `cc` / `c++` wrapper. It recognizes `cc`, `c++`, `gcc`, `g++`, `clang`, and `clang++` (plus versioned variants like `gcc-13` and target-prefixed cross compilers like `arm-linux-gnueabihf-gcc`), and `clang-cl` or any `--driver-mode=cl` invocation (clang in MSVC driver mode) — on every OS, not just Windows, since recognition keys off the driver mode rather than the host. For unrecognized compiler wrapper names (such as custom scripts, symlinks/copies of gcc, or thin shims), kache dynamically detects the compiler family via `-E` preprocessor probing. The wrapper executable itself must accept the usual preprocessor CLI (forwarding `-E` and defining `__clang__` or `__GNUC__`). Note that nested multi-token wrappers (e.g., `kache ccache gcc`) are not expanded. clang-cl is what mozconfigs and the `cc` crate use on Windows:

```sh
# POSIX (gcc / clang)
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/c-cpp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ For clang-cl / MSVC-driver mode:
set "CC=kache clang-cl"
```

kache recognizes `cc`, `c++`, `gcc`, `g++`, `clang`, `clang++`, versioned variants like `gcc-13`, target-prefixed cross compilers like `arm-linux-gnueabihf-gcc`, `clang-cl`, and `clang --driver-mode=cl`.
kache recognizes `cc`, `c++`, `gcc`, `g++`, `clang`, `clang++`, versioned variants like `gcc-13`, target-prefixed cross compilers like `arm-linux-gnueabihf-gcc`, `clang-cl`, and `clang --driver-mode=cl`. Any other compiler wrapper names that miss this allowlist are dynamically detected via `-E` preprocessor probing.

## What Is Cached

Expand Down
141 changes: 140 additions & 1 deletion src/compiler/cc.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3671,6 +3671,16 @@ fn named_tool_family(name: &str) -> Option<ToolFamily> {
})
}

fn is_unresolvable_bare_program(program: &str) -> bool {
if program.contains('/') {
return false;
}
if program.contains('\\') {
return false;
}
super::resolve_program_on_path(program).is_none()
}

impl CcCompiler {
pub fn new() -> Self {
Self::default()
Expand Down Expand Up @@ -3703,6 +3713,9 @@ impl CcCompiler {
/// Owns its own detection rule; `super::detect_compiler` reaches it
/// through this module's [`ADAPTER`] descriptor.
pub fn recognizes(args: &[String]) -> bool {
if super::is_workspace_wrapper_chain(args) {
return false;
}
let Some(arg0) = args.first() else {
return false;
};
Expand Down Expand Up @@ -3734,7 +3747,15 @@ impl CcCompiler {
return true;
}

false
// ── Slow path: `-E` probe for unknown binaries ──
if super::is_kache_subcommand_or_flag(&name) {
return false;
}
if is_unresolvable_bare_program(arg0) {
return false;
}

crate::probe::probe_compiler_family(arg0).is_some()
}

/// Does this argv match the `cc` Rust crate's compiler-family
Expand Down Expand Up @@ -4811,6 +4832,124 @@ mod tests {
}
}

#[test]
fn recognizes_unknown_wrapper_via_probe() {
if cfg!(target_os = "macos") {
return; // Apple's /usr/bin/cc re-dispatches on argv[0] via xcode-select
}

let _lock = crate::config::config_path_lock();
let temp = tempfile::TempDir::new().unwrap();
// Find a compiler on the system PATH to copy.
let compilers = ["cc", "gcc", "clang"];
let source_compiler = compilers.iter().find_map(|&c| {
let path = crate::compiler::resolve_program_on_path(c)?;
if crate::probe::probe_compiler_family(path.to_str()?).is_some() {
Some(path)
} else {
None
}
});
let Some(source_path) = source_compiler else {
return; // Skip if no GCC/Clang C compiler is installed.
};

// Copy or symlink it to an unrecognized name in temp directory.
let custom_name = if cfg!(windows) {
"my custom & compiler.cmd"
} else {
"my-custom-compiler"
};
let dest_path = temp.path().join(custom_name);

#[cfg(unix)]
{
std::os::unix::fs::symlink(&source_path, &dest_path).unwrap();
}
#[cfg(windows)]
{
std::fs::write(
&dest_path,
format!("@echo off\r\n\"{}\" %*", source_path.display()),
)
.unwrap();
}

// recognizes() should successfully probe and return true!
let dest_str = dest_path.to_str().unwrap().to_string();
assert!(CcCompiler::recognizes(std::slice::from_ref(&dest_str)));

// The same wrapper must be detected when it is found by PATH. This
// exercises the bare-name guard and the OS's safe argument handling
// for the Windows `.cmd` name containing spaces and `&`.
let previous_path = std::env::var_os("PATH");
let mut path_entries = vec![temp.path().to_path_buf()];
if let Some(previous) = previous_path.as_deref() {
path_entries.extend(std::env::split_paths(previous));
}
let joined_path = std::env::join_paths(path_entries).unwrap();
unsafe {
std::env::set_var("PATH", joined_path);
}
let recognized_by_bare_name = CcCompiler::recognizes(&s(&[custom_name]));
unsafe {
match previous_path {
Some(previous) => std::env::set_var("PATH", previous),
None => std::env::remove_var("PATH"),
}
}
assert!(recognized_by_bare_name);

// Must also succeed during actual wrapper dispatch when KACHE_ACTIVE is set in wrapper mode
let recognized_during_dispatch = {
let prev = std::env::var_os("KACHE_ACTIVE");
unsafe {
std::env::set_var("KACHE_ACTIVE", "1");
}
struct Guard(Option<std::ffi::OsString>);
impl Drop for Guard {
fn drop(&mut self) {
unsafe {
match self.0.as_ref() {
Some(val) => std::env::set_var("KACHE_ACTIVE", val),
None => std::env::remove_var("KACHE_ACTIVE"),
}
}
}
}
let _guard = Guard(prev);
CcCompiler::recognizes(std::slice::from_ref(&dest_str))
};
assert!(
recognized_during_dispatch,
"unknown compiler wrapper must be recognized during wrapper dispatch when KACHE_ACTIVE is set"
);
}

#[test]
fn recognizes_does_not_probe_kache_subcommands() {
assert!(!CcCompiler::recognizes(&s(&["list"])));
assert!(!CcCompiler::recognizes(&s(&["gc"])));
assert!(!CcCompiler::recognizes(&s(&["monitor"])));
assert!(!CcCompiler::recognizes(&s(&["config"])));
}

#[test]
fn recognizes_checks_path_separators_and_path_resolution() {
// Bare name not on PATH -> returns false without probing
assert!(!CcCompiler::recognizes(&s(&[
"kache_nonexistent_cc_binary_12345"
])));

// Path with separators that does not exist -> returns false
let nonexistent_path = if cfg!(windows) {
r"C:\nonexistent\path\to\mycc"
} else {
"/nonexistent/path/to/mycc"
};
assert!(!CcCompiler::recognizes(&s(&[nonexistent_path])));
}

#[test]
fn recognizes_family_probe_matches_dash_e_with_file_arg() {
assert!(CcCompiler::recognizes_family_probe(&s(&[
Expand Down
5 changes: 4 additions & 1 deletion src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1823,6 +1823,9 @@ pub(crate) fn parse_size_checked(value: &str, source: &str) -> Option<u64> {
parsed
}

#[cfg(test)]
pub(crate) use tests::config_path_lock;

#[cfg(test)]
mod tests {
use super::*;
Expand All @@ -1843,7 +1846,7 @@ mod tests {
);
}

fn config_path_lock() -> std::sync::MutexGuard<'static, ()> {
pub(crate) fn config_path_lock() -> std::sync::MutexGuard<'static, ()> {
static LOCK: OnceLock<Mutex<()>> = OnceLock::new();
LOCK.get_or_init(|| Mutex::new(())).lock().unwrap()
}
Expand Down
4 changes: 2 additions & 2 deletions src/daemon.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5589,8 +5589,8 @@ mod tests {
}
#[cfg(windows)]
{
std::process::Command::new("cmd")
.args(["/c", "ping", "-n", "31", "127.0.0.1"])
std::process::Command::new("ping")
.args(["-n", "31", "127.0.0.1"])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn()
Expand Down
5 changes: 5 additions & 0 deletions src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,11 @@ fn init_logging(mode: LogMode) {
}

fn main() -> Result<()> {
if std::env::var_os("KACHE_FAMILY_PROBE_ACTIVE").is_some() {
// Prevent unbounded recursion when a probed wrapper calls back into kache.
return Ok(());
}

let env_args: Vec<String> = std::env::args().collect();
let log_mode = detect_log_mode(&env_args);

Expand Down
14 changes: 14 additions & 0 deletions src/platform.rs
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,20 @@ pub fn kill_process(pid: u32) {
}
}

/// Forcefully kill a process and all its descendants (process group on Unix, process tree on Windows).
pub fn kill_process_group(pid: u32) {
#[cfg(unix)]
unsafe {
libc::kill(-(pid as i32), libc::SIGKILL);
}
#[cfg(windows)]
{
let _ = std::process::Command::new("taskkill")
.args(["/F", "/T", "/PID", &pid.to_string()])
.output();
}
}

#[cfg(windows)]
fn windows_terminate(pid: u32) {
use windows_sys::Win32::Foundation::CloseHandle;
Expand Down
60 changes: 47 additions & 13 deletions src/probe/cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,25 @@ pub fn probe_key(prober_id: &str, req: &ProbeRequest<'_>) -> Option<String> {
Some(h.finalize().to_hex().to_string())
}

/// A narrower key for probes that don't depend on the process environment or
/// arguments (e.g. `cc-family` detecting `__clang__` vs `__GNUC__`).
pub fn probe_key_isolated(prober_id: &str, program: &str) -> Option<String> {
let resolved = resolve_program(program)?;
let meta = std::fs::metadata(&resolved).ok()?;
let fingerprint = compiler_fingerprint(&meta);

let mut h = blake3::Hasher::new();
h.update(b"probe_schema:");
h.update(PROBE_SCHEMA_VERSION.to_string().as_bytes());
h.update(b"\nprober:");
h.update(prober_id.as_bytes());
h.update(b"\ncompiler_path:");
h.update(resolved.to_string_lossy().as_bytes());
h.update(b"\ncompiler_stat:");
h.update(fingerprint.as_bytes());
Some(h.finalize().to_hex().to_string())
}

/// Hash of the process environment — see [`fingerprint_env`].
///
/// `cc -###` inherits this environment, so a change to it (a different
Expand Down Expand Up @@ -81,18 +100,14 @@ fn fingerprint_env(vars: impl Iterator<Item = (String, String)>) -> String {
}

/// True for environment-variable names that vary between otherwise
/// identical build invocations and so must stay out of the probe key.
/// identical build invocations or are kache-internal/test environment
/// controls and so must stay out of the probe key.
///
/// Windows' process environment carries hidden cmd.exe bookkeeping
/// variables whose names begin with `=`: the per-drive working directory
/// (`=C:`, `=D:`, …) and the previous child's exit status (`=ExitCode`).
/// `std::env::vars()` surfaces them, yet they are never inputs to
/// `cc -###` and they shift between the cold and warm builds — which made
/// the on-disk probe memo miss on the warm build, re-running the probe
/// (#201). A Unix variable name cannot contain `=`, so this never fires
/// there.
/// This excludes kache-internal variables (`KACHE_*`) which configure kache
/// or tests, as well as Windows' hidden cmd.exe bookkeeping variables whose
/// names begin with `=` (`=C:`, `=ExitCode`, …).
fn is_volatile_env_name(name: &str) -> bool {
name.starts_with('=')
name.starts_with('=') || name.starts_with("KACHE_")
}

/// Load a probe record by key, or `None` on any miss: file absent,
Expand Down Expand Up @@ -268,13 +283,15 @@ mod tests {
}

#[test]
fn is_volatile_env_name_flags_only_equals_prefixed() {
fn is_volatile_env_name_flags_equals_and_kache_internal_vars() {
assert!(is_volatile_env_name("=C:"));
assert!(is_volatile_env_name("=ExitCode"));
assert!(is_volatile_env_name("KACHE_ACTIVE"));
assert!(is_volatile_env_name("KACHE_FAMILY_PROBE_ACTIVE"));
assert!(is_volatile_env_name("KACHE_CACHE_DIR"));
assert!(is_volatile_env_name("KACHE_CONFIG"));
assert!(!is_volatile_env_name("PATH"));
assert!(!is_volatile_env_name("SDKROOT"));
// A normal name that merely contains `=` later cannot occur, but
// guard the boundary: only a leading `=` is volatile.
assert!(!is_volatile_env_name("A=B"));
}

Expand Down Expand Up @@ -308,6 +325,23 @@ mod tests {
assert!(probe_key("cc", &req("/nonexistent/kache-cc-xyz")).is_none());
}

#[test]
fn probe_key_isolated_returns_valid_hash_and_bails_on_missing() {
let compiler = NamedTempFile::new().unwrap();
let path_str = compiler.path().to_str().unwrap();

let k1 = probe_key_isolated("cc-family", path_str).unwrap();
assert_eq!(k1.len(), 64, "Blake3 hex digest must be 64 characters");

let k2 = probe_key_isolated("cc-family", path_str).unwrap();
assert_eq!(k1, k2);

let k_other = probe_key_isolated("other-id", path_str).unwrap();
assert_ne!(k1, k_other, "different prober_id must yield different key");

assert!(probe_key_isolated("cc-family", "/nonexistent/kache-cc-xyz").is_none());
}

#[test]
fn with_exe_suffix_appends_when_set_and_absent() {
assert_eq!(
Expand Down
Loading