Skip to content

Commit af69e6d

Browse files
committed
tty: a latency is only an input's own answer; refuse AmbiguousInteraction otherwise, and start the launch clock after strobe's set-up
A terminal key sent while an earlier key's answer was still on its way took that answer for its own: an 80 ms answer read 24.7 ms (Windows), a 50 ms one 0.17 ms (Linux), printed as valid since 0.18.0. - Pairing: each driver call that sends input begins an action; an action sent while an earlier one (under 1 s before) had changed no cells, and then answered, overran, and so does every action after it until 1 s passes without one. Any overrun, burst input or kernel-echoed input refuses every tty.latency.* and tty.settle.* of the block AmbiguousInteraction, as does busy past half the samples; never the fast keys kept and the slow ones dropped. Busy is also judged from the grid after the run (an update spanning the input, a change under 50 ms before it). - Kernel echo: tcgetattr on the master before each input (ECHO, or ECHONL in canonical mode). - page.type waits up to 1 s for each paced key's answer (was 300 ms); a key that changes nothing waits the second and the next is still paced. type(..., { paste: true }) sends one paste (bracketed once the program set ?2004): one sample. press(key, { burst: true }) sends at once. - tty.latency.busy and .silent are values, 0 included, where the block has latency. - Launch clock: started after the pseudo-terminal, the console host and the reader are up; on Windows right before the suspended process resumes, on Unix right before the spawn. - A check skips cpu.instructions where only a counter block measures it (linux-promo-2). Windows: keys-80 back to back now refuses (was 23.3 ms, timed A/B +13% for +290%); spaced 20 -> 80 ms reads +59.99 ms, A/A same; ready-200 -> ready-500 +301.7 ms with A/A same (was A/A unclear, launches of 258-1632 ms). Linux: keys-50 refuses (was 0.35 ms, timed -98.8%), cat refuses ECHO (was 47 us); spaced +60.02 ms; ready +303.0 ms (cgroup set-up still inside the spawn).
1 parent 384805d commit af69e6d

4 files changed

Lines changed: 356 additions & 66 deletions

File tree

‎sdk/bench/src/index.ts‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -191,10 +191,19 @@ export interface Page {
191191
* (headed Chromium, headless pages). Pacing stops for the rest of the call once the target never rests (an animation).
192192
* `burst: true` sends the keys `delay` apart without waiting: no key is a latency sample (`interaction.*` refuses
193193
* `AmbiguousInteraction`); measure burst typing by `cpu.work`, `main.work`, `ui.cpu`, `run.duration` or a phase.
194+
* A terminal target (`tty()`) waits up to a second for each key's change (a key that changes nothing waits the
195+
* whole second), and `paste: true` sends the text as one paste instead (bracketed, `ESC [200~ … ESC [201~`, once
196+
* the program turned bracketed paste on): one `tty.latency` sample, to the paste's first change.
194197
*/
195-
type(selector: Element, text: string, opts?: Options & { delay?: number; burst?: boolean }): Promise<void>
196-
/** A key or chord: `"Enter"`, `"F9"`, `"Control+a"`, `"ControlOrMeta+Shift+P"`. `opts.selector` sends it to that element (clicked into focus first, as `type` does). */
197-
press(key: string, opts?: Options & { selector?: Element }): Promise<void>
198+
type(selector: Element, text: string, opts?: Options & { delay?: number; burst?: boolean; paste?: boolean }): Promise<void>
199+
/**
200+
* A key or chord: `"Enter"`, `"F9"`, `"Control+a"`, `"ControlOrMeta+Shift+P"`. `opts.selector` sends it to that element (clicked into focus first, as `type` does).
201+
* A terminal target (`tty()`) sends it once the grid held still 50 ms and does not wait for its answer: pass
202+
* `until` before the next input when the answer may take longer (else `tty.latency.*` refuses `AmbiguousInteraction`).
203+
* `burst: true` sends it at once (a chord's next key, `g` `g`), no latency sample: a block that sends one refuses
204+
* `tty.latency.*`.
205+
*/
206+
press(key: string, opts?: Options & { selector?: Element; burst?: boolean }): Promise<void>
198207
/**
199208
* Presses one key and holds it across other calls until `up`: a game's `"w"`, or `"Shift"` (held modifiers mark later keys).
200209
* A mouse button: `"Mouse"` (left), `"MouseRight"`, `"MouseMiddle"`, pressed where the pointer is, or first at `opts.at`

‎src/app/bench/tty.rs‎

Lines changed: 55 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,46 @@ fn host_cycles(tty: &Tty) -> u64 {
258258
/// What a terminal block measures beyond CPU and memory, said when a declared metric has no value.
259259
const MEASURES: &str = "run.duration, cpu.work, cpu.app.total, cpu.pty_host, cpu.wakes, memory.app.private, io.*, tty.latency.*, tty.settle.*, tty.updates, tty.frames, tty.frame.*, tty.bytes, tty.cells, count.*, phase.*, app.*";
260260

261+
/// Why the block's inputs have no latency strobe can pair with them (`AmbiguousInteraction`, which
262+
/// refuses every `tty.latency.*` and `tty.settle.*`), first reason first: its summary after the
263+
/// metric's name, and the fix.
264+
fn ambiguous(lat: &drv::Latencies) -> Option<(String, &'static str)> {
265+
let tried = lat.ms.len() + lat.busy;
266+
if lat.burst > 0 {
267+
Some((
268+
format!("the runs sent {} inputs with burst: true, each before the one before it answered, so no grid update is any one input's answer", lat.burst),
269+
"measure burst input by cpu.work, run.duration, tty.bytes or a phase, or drop burst: true (or use type with paste: true for one paste) for input latency",
270+
))
271+
} else if lat.echoed > 0 {
272+
Some((
273+
format!("the terminal's kernel line discipline had ECHO on for {} input(s), so the grid showed the kernel's echo before the program read the input", lat.echoed),
274+
"wait in setup until the program enters raw mode, or time a line-mode command's answer after Enter with a phase and until: \"text=…\"",
275+
))
276+
} else if lat.overran + lat.after_overrun > 0 {
277+
let ms = drv::ANSWER_MAX.as_millis();
278+
let what = match (lat.overran, lat.after_overrun) {
279+
(0, n) => format!("{n} input(s) came less than {ms} ms after inputs outside the measured runs (setup, a reset, a warmup run) that went out before an earlier one changed the grid"),
280+
(o, 0) => format!("{o} input(s) were sent while an earlier input within {ms} ms had not yet changed the grid"),
281+
(o, n) => format!("{o} input(s) were sent while an earlier input within {ms} ms had not yet changed the grid, and {n} more before {ms} ms passed without input"),
282+
};
283+
Some((
284+
format!("input outran the program ({what}), so late output cannot be paired with the input that caused it"),
285+
"wait for each input's answer before sending the next (pass { until: \"frame\" }, { until: \"idle\" } or { until: \"text=…\" } to page.press/click, or use paced page.type; after an input that changes nothing, page.wait(1000)), or measure throughput with cpu.work or run.duration",
286+
))
287+
} else if tried > 0 && lat.busy * 2 > tried {
288+
Some((
289+
format!(
290+
"the program was still drawing when {} of {tried} inputs were sent (not {} ms still after the last change), so the next update may not be their answer",
291+
lat.busy,
292+
drv::QUIET.as_millis()
293+
),
294+
"wait for the grid to settle before each input (until: \"idle\"), or stop spinners and animations in the target",
295+
))
296+
} else {
297+
None
298+
}
299+
}
300+
261301
/// One measured run.
262302
struct Run {
263303
ms: f64,
@@ -686,22 +726,32 @@ pub(super) fn chunk(ctx: &mut Ctx, cases: &[Series], v: &Variant, pair: usize, x
686726
let settled: Vec<metrics::Timed> = measured.iter().map(|r| metrics::Timed { t1: r.settled, ..r.timed.clone() }).collect();
687727
let lat = drv::latencies(&tty, &driver.inputs(), &settled.iter().collect::<Vec<_>>());
688728
let stretches: Vec<&metrics::Timed> = measured.iter().map(|r| &r.timed).collect();
729+
// Some input's latency is no input's answer for sure: none of the block's (never the
730+
// inputs strobe could pair alone, which would leave out the slow answers).
731+
let ambiguous = ambiguous(&lat);
732+
for m in declared_metrics(s).into_iter().filter(|m| m.starts_with("tty.latency.") || m.starts_with("tty.settle.")) {
733+
if let Some((summary, fix)) = &ambiguous {
734+
refused.insert(m.clone(), Refusal::new(ValidatorId::AmbiguousInteraction, format!("{m}: {summary}")).fix(*fix));
735+
}
736+
}
689737
let mut ms = lat.ms.clone();
690738
ms.sort_by(f64::total_cmp);
691-
if !ms.is_empty() {
739+
if !ms.is_empty() && ambiguous.is_none() {
692740
values.insert("tty.latency.n".into(), ms.len() as f64);
693741
values.insert("tty.latency.p50".into(), metrics::percentile(&ms, 0.5));
694742
values.insert("tty.latency.p95".into(), metrics::percentile(&ms, 0.95));
695743
}
696744
let mut settle = lat.settle_ms.clone();
697745
settle.sort_by(f64::total_cmp);
698746
// A resize after which only the host drew would leave its slowest settles out: none then.
699-
if !settle.is_empty() && lat.host_only == 0 {
747+
if !settle.is_empty() && lat.host_only == 0 && ambiguous.is_none() {
700748
values.insert("tty.settle.n".into(), settle.len() as f64);
701749
values.insert("tty.settle.p50".into(), metrics::percentile(&settle, 0.5));
702750
values.insert("tty.settle.p95".into(), metrics::percentile(&settle, 0.95));
703751
}
704-
if lat.busy + lat.silent > 0 {
752+
// The inputs that were no sample, by why: counts, 0 included (a declared one is a value then,
753+
// not a refusal), none where the block's latency is refused.
754+
if ambiguous.is_none() {
705755
values.insert("tty.latency.busy".into(), lat.busy as f64);
706756
values.insert("tty.latency.silent".into(), lat.silent as f64);
707757
}
@@ -739,7 +789,8 @@ pub(super) fn chunk(ctx: &mut Ctx, cases: &[Series], v: &Variant, pair: usize, x
739789
// What this machine cannot read is said as that, before what the block did not measure.
740790
refuse_not_on_this_os(s, &mut values, &mut refused);
741791
for m in declared_metrics(s) {
742-
if values.contains_key(m) || refused.contains_key(m) || m.starts_with("count.") || m.starts_with("phase.") {
792+
// A counter only a counter block measures here is that block's to give (none in a check).
793+
if values.contains_key(m) || refused.contains_key(m) || m.starts_with("count.") || m.starts_with("phase.") || (!x.counters && super::counted_apart(m)) {
743794
continue;
744795
}
745796
let r = match m.as_str() {

0 commit comments

Comments
 (0)