RFC 211: Support testdriver.js in other test types

[jonathan-j-lee]

Rendered

Tracking issue: web-platform-tests/wpt#13183
Prototype implementation: web-platform-tests/wpt#48486
CC: @mfreed7 @WeizhongX

[WeizhongX]

LGTM

[chrishtr]

@jgraham could you review? We need this in order to test customizable select menus, among other things. Thanks!

[past]

Here are the relevant meeting notes from this week's discussion.

[jgraham]

So, in principle I support the idea of this. In practice it's really hard for Gecko to implement at the moment, because we have a specific internal endpoint for running reftests, and although it's not impossible to imagine a mechanism to allow it to be interrupted, it feels likely to be challenging. In theory we could fall back to the more standard reftest implementation only for these reftests, allowing an implementation much more similar to the WebDriver one, although that might represent a surprising amount of behaviour change.

Also in the meeting @gsnedders brought up some concerns that even with this support some of the use cases listed might be challenging to support cross-browser. In particular, testing <select> might end up in poorly-defined territory; whether the popup window should be accessible to WebDriver seems unclear, although maybe the customizable select work itself makes that clearer.

[WeizhongX]

So, in principle I support the idea of this. In practice it's really hard for Gecko to implement at the moment, because we have a specific internal endpoint for running reftests,

I thought with this change, reftests do not use testdriver should continue to work as usual?

Jonathan has a draft PR here: web-platform-tests/wpt#48486.

Also in the meeting @gsnedders brought up some concerns that even with this support some of the use cases listed might be challenging to support cross-browser. In particular, testing <select> might end up in poorly-defined territory; whether the popup window should be accessible to WebDriver seems unclear, although maybe the customizable select work itself makes that clearer.

Maybe in this RFC our goal is to enable testdriver for reftests, crash tests etc? Other issues should be addressed separately. Popup window is an issue even for now, right?

[mfreed7]

Also in the meeting @gsnedders brought up some concerns that even with this support some of the use cases listed might be challenging to support cross-browser. In particular, testing <select> might end up in poorly-defined territory; whether the popup window should be accessible to WebDriver seems unclear, although maybe the customizable select work itself makes that clearer.

Just to clarify, there are two "modes" that <select> can use for popups. One is the old/existing one, where (in all mainstream browsers) the popup window is actually a separate OS window. I agree that making this interoperably-testable will be tough. However, the other mode is the new "customizable-<select>" mode, where select, ::picker(select) {appearance: base-select} is used. In that case, the picker is defined to be a popover, so it renders right in the same renderer. There are no separate popup windows. That mode is the one for which we badly need this capability. And hopefully it doesn't present any issues, right?

[jonathan-j-lee]

Edited the "Risks" section a bit to capture the practical implementation challenges discussed. PTAL @jgraham @gsnedders

[gsnedders]

I think it's true that the only real risk here is with the internal Marionette runner; otherwise this has broadly seemed like a good idea for a while.

[jgraham]

I think this is OK; a Gecko implementation that falls back to the WebDriver-style external runner isn't ideal, but it's likely better than not having the feature at all.

[jonathan-j-lee]

@web-platform-tests/wpt-core-team Ready to land?

[past]

Thanks!