[WebDriver BiDi] emulate user-agent string

[sadym-chromium]

Support for user-agent emulations. Required for WebDriver BiDi: https://www.w3.org/TR/webdriver-bidi/#command-emulation-setUserAgentOverride (w3c/webdriver-bidi#448).

Introduce "environment default User-Agent value" to reference it from the HTML specification.

• At least two implementers are interested (and none opposed):
  • Chromium
  • Mozilla
• Tests are written and can be reviewed and commented upon at:
  • [wdspec] tests for emulation.setUserAgentOverride command web-platform-tests/wpt#54810
• Implementation bugs are filed:
  • Chromium: feat: emulation.setUserAgentOverride GoogleChromeLabs/chromium-bidi#3661
  • Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1987935
  • WebKit: N/A
  • Deno (not for CORS changes): N/A
• MDN issue is filed: N/A
• The top of this comment includes a clear commit message to use.

(See WHATWG Working Mode: Changes for more details.)

---

Preview | Diff

[sadym-chromium]

@noamr PTAL

[noamr]

Unofficial LGTM

[annevk]

I had these comments, but you just refactored it further. I'm not sure I like removing the default User-Agent value concept entirely. Is that really what we want?

I now also wonder at what point implementations ask WebDriver for this value as it seems unlikely to happen in het network layer, where the User-Agent value is currently appended.

[sadym-chromium]

I had these comments, but you just refactored it further. I'm not sure I like removing the default User-Agent value concept entirely. Is that really what we want?

I now also wonder at what point implementations ask WebDriver for this value as it seems unlikely to happen in het network layer, where the User-Agent value is currently appended.

@annevk Ok, I reverted the last change where I merged the "environment default <code>User-Agent</code> value" and "default <code>User-Agent</code> value" and applied your proposed changes.

[annevk]

So this looks good, but what about the timing of this call? Presumably it happens earlier, but maybe we can decide not to worry about that for now? Hmm.

[sadym-chromium]

I now also wonder at what point implementations ask WebDriver for this value as it seems unlikely to happen in het network layer, where the User-Agent value is currently appended.

@annevk in Chromium, we rely on Emulation.setUserAgentOverride method, which sets the user agent override, which is read in the renderer.

[annevk]

I think that does mean there are some issues with this setup. In that if you hit a couple of redirects and you call that method repeatedly, the User-Agent header would not change in value, but per this specification as written it ought to change.

To fix that and some race issues we'd have to grab the current User-Agent header before we go in parallel and use it for the entire duration of the fetch. I'm okay with not doing that for now, but I'm curious what other people think about what level of accuracy we need here. Maybe @jgraham is willing to chime in.

[sadym-chromium]

I think that does mean there are some issues with this setup. In that if you hit a couple of redirects and you call that method repeatedly, the User-Agent header would not change in value, but per this specification as written it ought to change.

To fix that and some race issues we'd have to grab the current User-Agent header before we go in parallel and use it for the entire duration of the fetch. I'm okay with not doing that for now, but I'm curious what other people think about what level of accuracy we need here. Maybe @jgraham is willing to chime in.

The WebDriver BiDi "WebDriver BiDi emulated User-Agent" hook returns the emulated user context based on pre-configured emulations per top-level navigable, so it is not expected to be changed during navigations or redirects. Even though the JS environments will be changed, it would not influence the return result. BiDi uses the environment only to associate the request with the navigable.

[annevk]

I see. So you are you saying it's set before navigation starts and can't be changed after?

[sadym-chromium]

I see. So you are you saying it's set before navigation starts and can't be changed after?

Theoretically the user can change the emulation during navigation, but otherwise it is persistent per top-level navigable. I cannot imagine a real-world automation scenario when the user needs to change emulation during navigation and expect persistent results.

[jgraham]

You can change it in the middle of a fetch, so I think it's true that redirects could see different user agent headers (you could also do the same thing with request interception).

In general I don't think people are expected to change this often for existing traversables; the typical use case is "set the user agent when creating a top-level traversable and use it for the entire lifetime of that traversable".

So I think I agree that in principle caching the value at the start of a redirect loop is the right thing to do, but in practice it's likely to be a rare edge case.

[annevk]

Since WebDriver interoperability is quite a bit less crucial than web platform interoperability I'll go ahead and merge this. Let's revamp if people end up hitting this in the wild after all.

[sadym-chromium]

Since WebDriver interoperability is quite a bit less crucial than web platform interoperability I'll go ahead and merge this. Let's revamp if people end up hitting this in the wild after all.

Thanks a lot!