Collecting compat data for WebDriver(-BiDi)

[Elchi3]

Opening this issue to discuss whether it would be possible for us to collect compatibility information for WebDriver-(BiDi).

cc @whimboo

BCD issue: mdn/browser-compat-data#20207
WebDriver-Bidi issue: w3c/webdriver-bidi#619

Currently, the data lives in a spreadsheet: https://docs.google.com/spreadsheets/d/1bkiPU5eDBCqFkx5p_VSBx_OK8gy9TeHRKQVPHKMATGQ/edit?pli=1&gid=1960491400#gid=1960491400

As usual, we need two things, (1) we need a way to get to know about (new) features and (2) we need a way to test these features using minimal test code to determine if a feature works in a browser.

(1) Getting to know about features

For Web APIs, we find out about new API via webref/idl files and our custom idls. For WebDriver-Bidi there seem to be CDDL files analog to this.

• They can be generated from the spec at https://github.com/w3c/webdriver-bidi.
• Here's a gist that shows an example https://gist.github.com/Elchi3/13c08b12773dc5b5a58e5cf19d0f4238
• There is a CDDL parser: https://www.npmjs.com/package/cddl
• Or you can get TypeScript directly: https://www.npmjs.com/package/cddl2ts
• With this information we could generate a list of BCD features as discussed in Generate browser-compat-data from CDDL definition w3c/webdriver-bidi#619

Q: Is the same available for WebDriver-classic or should we have a custom list of features (which would live in https://github.com/openwebdocs/mdn-bcd-collector/tree/main/custom)?
Q: If we don't want to get in the business of parsing CDDLs now, we could make custom list of features which will have to update manually as the spec changes.

(2) Testing WebDriver features and determining version numbers

Once we have (1) in place, we know which sorts of webdriver(-bidi) features exist and now we need to collect compat information, that is: in which browser versions do we have support for these features?

• See https://wpt.fyi/results/webdriver for WPT tests. (they are probably not minimal enough, though, according to Henrik)
• Need to create a new test builder for webdriver in https://github.com/openwebdocs/mdn-bcd-collector/tree/main/test-builder
• Might need custom tests for certain WebDriver features

Q: Which sort of tests come to mind here? I'm assuming there is no simple way to just call a method like navigator.webdriver.supports('foo') (fantasy API, analog to CSS.supports('foo')) and determine support? How would we tests WebDriver features?

Once (2) is in place as well, we can create BCD features with compatibility data collected and we can then send that to the BCD repository on a regular basis.

[whimboo]

Thanks for filing the issue @Elchi3! I'm looking forward to collaborate for getting these information into BCD. For now just some comments on your previous comment:

They can be generated from the spec at https://github.com/w3c/webdriver-bidi.

There is no need to write code to extract the CDDL from the specification. That's already done and runs automatically for each merge. We upload the generated files (client, browser) as artifact on the Test action like that one: https://github.com/w3c/webdriver-bidi/actions/runs/10265666158. It means you would only have to check if that file has changed and re-download for processing.

Q: Is the same available for WebDriver-classic or should we have a custom list of features (which would live in https://github.com/openwebdocs/mdn-bcd-collector/tree/main/custom)?

No, we don't have any parsable data in WebDriver classic. As such I would suggest that we just focus on BiDi for now which is more important for automation client authors and testers. Classic has pretty mach the same support across browsers these days.

Q: If we don't want to get in the business of parsing CDDLs now, we could make custom list of features which will have to update manually as the spec changes.

This might be quite an effort depending on how much the spec changes between various browser releases. It should be actually easy to convert the CDDL data to JSON given that there is also a library for Rust which uses Serde under the hood: https://github.com/anweiss/cddl. That means the structure can easily be read and exported as JSON. But BCD requires a specific format so some transformation would be required.

Q: Which sort of tests come to mind here? I'm assuming there is no simple way to just call a method like navigator.webdriver.supports('foo') (fantasy API, analog to CSS.supports('foo')) and determine support? How would we tests WebDriver features?

We probably should be able to create minimized tests for the API itself and its various parameters. For any API that is not implemented WebDriver raises a unsupported command error. That should basically apply as well to parameters. In case we need to update the parameter type, which could happen as well, and we test with a wrong type then WebDriver raises a invalid argument error. Maybe those simple tests might be enough?

[whimboo]

I just wanted to mention this blog post from WebDriverIO especially around the two packages to read the CDDL files into an AST: https://webdriver.io/de/blog/2023/06/08/bidi-typescript-support/

[whimboo]

Hi @Elchi3. Is there anything else that needs to be clarified and I can help with? I want to make sure that nothing is blocked based on incomplete information. If it's all fine do you maybe have a rough timeline of when work on that issue could be started? Thanks!

[caugner]

FWIW We cannot currenlty use the cddl NPM package, because it fails to parse the webdriver-bidi CDDL file: christian-bromann/cddl#107

[Elchi3]

(1) Getting to know about features

FYI w3c/webref#1353 is about doing this work in webref (which would be better as other webref consumers would then also benefit from it and we could get the necessary info via webref as we usually do, too).