Intermittent timeouts in Safari

[jugglinmike]

The nascent Mac Mini/Safari setup has demonstrated its first instability 🎉

wptrunner occasionally times out while running WebDriver specification tests in Safari and Safari Technology Preview. (While we've been testing Safari for over a year, we haven't yet experienced this because that testing has occurred on Sauce Labs. The WPT CLI skips "wdspec" tests when running on that service.)

The timeouts occur between tests (not during), and while no single test is responsible, all the timeouts occur immediately following a test concerning user prompts.

I can reproduce this via direct invocations of wpt run in about 10% of trials. This makes it likely to interfere with many collection attempts due to our sub-optimal segmenting strategy (see gh-614). That's mostly a concern for me because it means more manual intervention--I'm hopeful that I can maintain reliable results until we get this fixed. (Though today's collection will be delayed by ~24 hours because I didn't schedule the "rescue" builds in time.)

@jimevans Does this timeout sound familiar to you? It seems distinct from the TAB key timeouts you were describing in IRC the other day.

[jugglinmike]

This problem continues to interfere with results collection from Safari. I'm still researching the root cause.

[gsnedders]

One question is why are we not recovering from this? Because destroying/creating a new WebDriver session doesn't restart Safari and there's some shared state?

[jgraham]

Honestly if it's just a small number of known tests then we probably just want to disable them for now.

The wdspec tests share a session for performance reasons (each new session typically involves restarting the whole browser so is slow). That does mean that you can get cascading failures if the session gets into a state where it can't accept new commands. Typically we'd recover by noticing the unexpected failure and then restarting the runner. But of course since we don't have expectation data we don't do that on wpt.fyi.

[jugglinmike]

This appears to be an issue in wptrunner, though the conditions that cause it only occur when testing Safari (today, at least). More info here: web-platform-tests/wpt#13446 and a proposed fix here: web-platform-tests/wpt#13447

[jugglinmike]

We merged web-platform-tests/wpt#13447 to address this, and the subsequent collections for Safari and Safari Technology Preview both completed successfully.