Webdriver tests

Regression tests are run with webdriver. They compare actual screenshots of various test scenes to expected ones recorded at a previous release.

What are those tests?

These tests consist of two test files:

• One test file runs in a browser (either a local PhantomJS instance, or on different browsers provided by Sauce Labs).

• The other test file runs in node and actually "drives" the browser - opens urls, clicks on things in the window and takes screenshots. The latter ("webdriver") test file makes certain assertions, making the tests pass or fail (in our case we mostly compare screenshots).

Where do I insert new test cases?

• Look at tests/webdriver/index-webdriver.html. Insert <my-test-name> at the start of the file (you will see comment). Optionally specify a condition when it should be executed - you don't want to run WebGL tests in old browsers.

• For creating the two needed test files you have two available alternatives:

  • Statically: Create two test files, one named tests/webdriver/<my-test-name>.html and the other one named tests/webdriver/<my-test-name>.js (the former being the code that will run in browser, the latter being the webdriver code). Copy & paste existing test files (e.g. template-crafty.html & template-crafty.js) in order to ensure a base structure, that the test internals rely on.
  • Dynamically: Create a test file tests/webdriver/<my-test-name>.js that contains both the webdriver code and the code that can generate the test script which will run in a browser. Copy & paste existing test files (e.g. the color tests) in order to ensure a base structure, that the test internals rely on.

• When you first run your new tests via npm test (== grunt check), or more specifically via grunt test-local-webdriver, you will probably see error messages complaining about non-existent screenshots. This works as intended: The test internals look for corresponding screenshots in tests/assets/webdriver/<my-test-name>-expected.png. If they can't find them, they will be regenerated and error thrown. A second test run should give no more errors. Deleting existing expected screenshots is also possible. The expected screenshots are committed to the repository.

• Failed screenshots from local tests can be observed in the build/webdriver/failed directory. Failed screenshots from SauceLabs tests can be observed on Crafty Distro#regression-tests.

Anything to keep in mind when constructing a browser test page?

• Look at existing test pages.

• Preserve the base structure given by other test pages. Crafty will be initted for you with a resolution of 320 x 240 px, in the default cr-stage div. Background will be set to gray. CSS rules will remove all margins and paddings.

• This means all elements in test page should fit into the given resolution! If you want more freedom, checkout template-generic and template-local(.html & .js).

Ok, I constructed a browser test page. Which webdriver test commands can I use?

• Look at existing test cases. Don't forget the QUnit.module preamble.

• There is a global browser instance which you send commands to. The commands are all asynchronous and chain together like promises - don't forget to return the promise chain! All available commands are listed in WebdriverIO API docs.

• Additional commands are added by the test internals, which should be preferred to the WebdriverIO ones - they are specifically tailored to the (sometimes extreme) differences to features available in different browser drivers:

  • testUrl - navigates to proper test page
    • if no param provided, navigates to url matching your <my-test-name>
    • else if at least one param is provided, dynamically generates a test page and navigates to it (as explained above)
      • param of type string - serialized test script which will be inserted into dynamically generated test page <my-test-name>.html
      • optional 1st param of type string - <other-test-name> to use for dynamically generating the test page <other-test-name>.hml
  • assertResemble - compares newly taken screenshots to expected ones, or records new expected screenshots (as explained above)
    • optional param of type string - screenshot file-name modifier
      • param starts with - - prefix for the screenshot file-name, enables multiple screenshots per test file
      • param does not start with - - force specific screenshot file-name, enables comparison of one screenshot across multiple test files
    • optional param of type number - error threshold for image comparison, omit unless there is reason to increase it
    • optional param of type boolean - enable strict antialiasing image comparison, screenshots on android have to be stretched, resulting in antialiased edges, which are ignored by default - enable for precise image comparison (e.g. pixelart)
    • optional param of type object - crop the screenshot to specific region, defaults to whole 320x240 viewport
  • pointerMove(x,y) - move mouse or touch input to specified location
  • pointerDown() - press mouse or touch at previously moved-to location
  • pointerUp() - depress mouse or touch at previously moved-to location
  • keyDown(key) - trigger single keydown
  • keyUp(key) - trigger single keyup

What do waitBarrier and signalBarrier do?

These two functions serve for synchronization points between the test page and your webdriver test.

The test page has to somehow signal when it is done rendering all relevant Crafty entities - that's what signalBarrier(<label>) is for. You can then wait for a specific synchronization point with waitBarrier(<label>) in your webdriver test code - you can also increase the default wait timeout by passing an additional argument representing the timeout to wait in ms.

You can analogously wait for the webdriver test code to take a screenshot and signal its completion before proceeding to render something different on the test page.

Miscellaneous

• The file supported-browsers.json lists the browser we test on
• The permanent test fixtures are saved to tests/assets/webdriver and temporary test results to build/webdriver.
• All open sauce (in the cloud) tests only work on pre-release testing branch.
• Screenshots of failed cloud tests are uploaded to Crafty-Distro/regression-tests.
• In order to make it work for all browsers on Open Sauce, some dirty hacks have been included:
  • simulate synthetic events for those that do not support native ones
  • rotate, crop and scale screenshots from android browsers

Example

template.html - browser test page

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <title>CraftyJS Webdriver Test</title>
  <link rel="stylesheet" href="common.css">
  <script src='common.js'></script>
  <script src='../../crafty.js'></script>
</head>
<body>
<script>
window.addEventListener('load', function() {
  // ADD YOUR TEST CODE HERE; Crafty init'd with {w: 320, h: 240}

  // green floor
  Crafty.e("2D, DOM, Color, ground")
    .attr({ x: 0, y: 239, w: 320, h: 1 })
    .color('rgb(0,255,0)');

  // green platform
  Crafty.e("2D, DOM, Color, ground")
    .attr({ x: 250, y: 120, w: 60, h: 1 })
    .color('rgb(0,255,0)');

  // blue player
  var landedCount = 0;
  Crafty.e("2D, DOM, Color, Gravity, Twoway, Button, Draggable")
    .attr({ x: 265, y: 140, w: 30, h: 30 })
    .color('rgb(0,0,255)')
    .gravityConst(1 * 50 * 50)
    .twoway(5 * 50,17 * 50)
    .gravity("ground")
    .bind("StartDrag", function() {
      this.antigravity();
    })
    .bind("StopDrag", function() {
      this.gravity();
    })
    .bind("LandedOnGround", function(ground) {
      landedCount++;
      signalBarrier("landed" + landedCount);
    });

}, false);
</script>
</body>
</html>

template.js - node webdriver test

QUnit.module(module);

QUnit.test("Multi test template part 1: Player lands on ground floor automatically after navigating to test page", function(assert) {
  return browser
    .testUrl()
    .waitBarrier("landed1")
    .assertResemble("-firstLand", 0.20);
});

QUnit.test("Multi test template part 2: Player lands on platform after jump key pressed", function(assert) {
  return browser
    .keyDown('W').keyUp('W')
    .waitBarrier("landed2", 2000)
    .assertResemble("-secondLand", 0.20);
});

QUnit.test("Multi test template part 3: Player lands on floor after being drag & dropped", function(assert) {
  return browser
    .pointerMove(280, 107)
    .pointerDown()
    .pointerMove(25, 135)
    .pointerUp()
    .waitBarrier("landed3", 2000)
    .assertResemble("-thirdLand", 0.20);
});

Example screenshots

Produces in step 1 this screenshot

Produces in step 2 this screenshot

Produces in step 3 this screenshot

And checks if the new screenshots match the previously recorded ones within margin of error.