Dojo Automated Testing Improvements: Updating to Intern

By on February 18, 2014 6:49 am

2013 saw the development, release, and adoption of Intern, SitePen’s modern, standards-based JavaScript testing framework. One of our priorities for 2014 is converting Dojo’s automated tests from DOH to Intern. Although DOH was an advanced testing framework when it was released, the world of web development has seen some important improvements that necessitated the development of a new testing framework:

  • Remote automated multi-platform browser testing from services like Sauce Labs
  • WebDriver, a standard API for controlling web browser behavior, with a cross-browser implementation by Selenium
  • Continuous integration testing for JavaScript with Travis CI
  • Widespread adoption of server-side JavaScript with NodeJS

Taking advantage of these resources enables us to ensure that tests are run for all supported branches of Dojo, to assess the level of code coverage our tests provide, and to keep up to date with functional browser testing as vendors continue to update and improve web browsers. Updates to web browser security models for Java have made the DOH Java-based Robot functional tests impractical to continue running, and of course do not work on mobile platforms. Continuous integration testing allows us to more quickly integrate pull requests and reduce regression bugs, and maintain many more releases.

As an open source project, Dojo relies on the contributions of its users to keep up-to-date. Although SitePen is dedicating resources to updating Dojo’s tests, we hope that you will join us, improving our code coverage and ensuring that each release is more stable and reliable than the previous. With that in mind, we hope you will find the following information about creating tests with Intern helpful.

Creating tests with Intern

There are two main categories of testing for a project like Dojo: unit and functional.

  • Unit testing relates to modules that can be run strictly via JavaScript with no user interaction required.
  • Functional testing relates to modules that depend on user interaction with the web browser, such as mouse and keyboard events.

Unit tests tend to be the simplest to write, so we will start with an example of converting an existing DOH test to an Intern test. Intern’s wiki includes information on writing tests with Intern. Let’s look at conversion of the test for the dojo/store/Memory module:

(The old tests, excerpted from dojo/tests/store/Memory.js):

define(["doh", "dojo/store/Memory"], function(doh, Memory){
    doh.register("dojo.tests.store.Memory",
        [
            function testGet(t){
                t.is(store.get(1).name, "one");
                t.is(store.get(4).name, "four");
                t.t(store.get(5).prime);
            },
            function testQuery(t){
                t.is(store.query({prime: true}).length, 3);
                t.is(store.query({even: true})[1].name, "four");
            },
            function testQueryWithSort(t){
                t.is(store.query({prime: true}, {sort:[{attribute:"name"}]}).length, 3);
                t.is(store.query({even: true}, {sort:[{attribute:"name"}]})[1].name, "two");
                t.is(store.query({even: true}, {sort:function(a, b){
                        return a.name < b.name ? -1 : 1;
                    }})[1].name, "two");
                t.is(store.query(null, {sort:[{attribute:"mappedTo"}]})[4].name, "four");

                t.is([ 1, 5, 4, 2, 3 ], store.query({}, { sort: [ { attribute: "date", descending: false } ] }).map(function (item) {
                    return item.id;
                }));
            },
            function testAddNewIdAssignment(t){
                var object = {
                    random: true
                };
                store.add(object);
                t.t(!!object.id);
            }
        ]
    );
});

The same tests using Intern:

define([
    'intern!object',
    'intern/chai!assert',
    'dojo/store/Memory'
], function (registerSuite, assert, MemoryStore) {
    registerSuite({
        name: 'dojo/store/Memory',

        '.get': [
            function () {
                assert.strictEqual(store.get(1).name, 'one');
                assert.strictEqual(store.get(4).name, 'four');
                assert.ok(store.get(5).prime);
            }
        ],

        '.query': {
            'with boolean': function () {
                assert.strictEqual(store.query({ prime: true }).length, 3);
                assert.strictEqual(store.query({ even: true })[1].name, 'four');
            },

            'with sort': function () {
                assert.strictEqual(store.query({ prime: true }, { sort: [ { attribute: 'name' } ] }).length, 3);
                assert.strictEqual(store.query({ even: true }, { sort: [ { attribute: 'name' } ] })[1].name, 'two');

                assert.strictEqual(store.query({ even: true }, { sort: function (a, b) {
                    return a.name < b.name ? -1 : 1;
                }})[1].name, 'two');

                assert.strictEqual(store.query(null, { sort: [ { attribute: 'mappedTo' } ] })[4].name, 'four');

                assert.deepEqual(store.query({}, { sort: [ { attribute: 'date', descending: false } ] }).map(function (item) {
                    return item.id;
                }), [ 1, 5, 4, 2, 3 ]);
            }
        },

        '.add': {
            'new id assignment': function () {
                var object = {
                    random: true
                };

                store.add(object);
                assert.isDefined(object.id);
            }
        }

    });
});

Some things to note about the Intern tests:

  • As with DOH, test functions can be specified as an array or as named properties on an object. Named tests are preferred for providing clear test report information.
  • Tests can be specified in a nested structure for logical grouping and clear output. This is the console output in Google Chrome for the dojo/store/Memory tests:
    MemoryStoreTestResults
  • Tests that apply to a specific method should be grouped under the method name prefixed with a dot, a naming convention we have chosen to make it clear when a test name is related to a particular method. In this example, there is only a single test for the “get” method, so it was grouped under the “.get” property but specified as an array with a single element.
  • The Chai assertion library used by Intern has a variety of assertion methods, so read the reference documentation so you know how to use assert.isUndefined, assert.deepEqual and many others.
  • This is a great opportunity to not only update tests from DOH to Intern, but also to improve the quality and readability of the test code. There are also some modules with no existing test coverage.
  • We are using the coding conventions planned for Dojo 2, so be sure to read the Dojo 2 code conventions and adhere to them.

To coordinate collaboration on tests we have created a Dojo Toolkit Intern Conversion document where you can volunteer for tests you would like to convert. There are tabs at the bottom of that document for both Dijit and DojoX as well – feel free to populate them with information as needed for tests you would like to convert.

Contributing your tests

SitePen’s Bryan Forbes is coordinating the process of Intern adoption. Pull requests should be submitted to the “intern-conversion” branch of his Dojo fork.

Setting up your test environment

  1. On GitHub, create your own fork of Dojo from the main repository:
    https://github.com/dojo/dojo
  2. Clone your fork:
    git clone git@github.com:yourusername/dojo.git
  3. Add Bryan Forbes’ fork as a remote:
    git remote add bryanforbes https://github.com/bryanforbes/dojo.git
  4. Fetch Bryan Forbes’ repo:
    git fetch bryanforbes
  5. Create a branch based off of the “intern-conversion” branch:
    git checkout -b intern-conversion bryanforbes/intern-conversion
  6. Push this branch to your personal fork of Dojo on GitHub:
    git push --set-upstream origin intern-conversion
  7. Install Intern and its dependencies by running the following command in the dojo directory:
    npm install intern
  8. If you don’t already have Grunt installed, do so now:
    npm install -g grunt-cli

At this point you have the existing conversion work cloned to your local repository and checked out, Intern setup to run tests, and Grunt installed to automate your test workflow. The following steps will allow you to begin testing:

Running tests

  • Local browser testing:
    1. Start a local server for testing by running
      grunt test:proxy
    2. Run all tests by opening your browser to:
      http://localhost:9001/__intern/client.html?config=tests-intern/intern
    3. Run individual tests by specifying the module id as the “suites” URL parameter:
      http://localhost:9001/__intern/client.html?config=tests-intern/intern&suites=tests-intern/unit/string
  • Local Selenium server testing (Selenium must be installed, configured, and launched separately):
    • Run all tests against the local Selenium instance by running
      grunt test:local
  • Run tests remotely using Sauce Labs:
    grunt test
  • To get code coverage information, append “:coverage” to the test command:
    grunt test:coverage
  • Code coverage for local tests is output to the “dojo/html-report” directory
    grunt test:local:coverage

Creating and submitting tests

  1. Create a branch for each test suite based on bryanforbes/intern-conversion:
    git checkout -b dom-form-tests bryanforbes/intern-conversion
  2. Add your tests to the appropriate file and folder in the “dojo/tests-intern” folder (unit tests in the “unit” folder, functional tests in the “functional” folder)
  3. Run your tests and ensure they pass
  4. After you commit your changes, push the branch to your fork of Dojo on GitHub:
    git push origin dom-form-tests
  5. View your branch on GitHub: https://github.com/yourusername/dojo/tree/dom-form-tests
  6. Click the “Pull Request” link (above the file list, on the right)
  7. Above the comment entry area is a listing of which repository and branch you are submitting a pull request to. Click the “Edit” button on the right:
    1. base fork: bryanforbes/dojo
    2. base branch: intern-conversion
    3. head fork: <your personal fork of Dojo>
    4. compare/head branch: <your test branch>
  8. Click the “Files Changed” tab below the comment entry area to review the changes you are making a pull request for
  9. Enter a comment for the pull request and click the “Send pull request button”
  10. Thank you for contributing to Dojo!

New test branches will be merged into Bryan Forbes’ “intern-conversion” branch as pull requests are submitted, so in order to keep your “intern-conversion” branch in sync you should periodically update it:

  1. git fetch bryanforbes
  2. git checkout intern-conversion
  3. git rebase bryanforbes/intern-conversion

 Additional information

Travis CI test results

With each pull request, the test suite, including your changes, will be run against Travis-CI using Sauce Labs on each supported platform, to make sure that your tests all pass.
https://travis-ci.org/bryanforbes/dojo/builds/15997208

Intern test directory structure

The following structure is used for creating these tests:

tests-intern/
    intern.js - SauceLabs configuration
    intern.local.js - Local Selenium configuration
    functional/ - Functional tests
        all.js - Module referencing all functional tests to run
    unit/ - Unit tests
        all.js - Module referencing all unit tests to run
    support/ - Supporting files such as pre-run scripts for SauceLabs
    services/ - Service proxy server plus service modules

 Creating services

Most tests should be mocking their data instead of using live data from a service. Services should only be written for tests that absolutely MUST communicate with a server, such as tests for IO functionality.

If tests are required to communicate with a server (for instance, dojo/request), services can be written for the test runner. Simply create an AMD module and be sure the filename ends in “service.js”. The module must return a function that accepts a JSGI request object and returns either an object describing the response or a promise that will resolve to an object describing the response. For services, when.js was chosen for generating and manipulating promises, mostly because it is AMD compatible, but it also comes with a full host of utilities.

You can see services in action in the dojo/request tests (“tests-intern/unit/request”), with the associated services in “tests-intern/services/request”.

Services can be accessed from the proxy via the following URL pattern:
http://localhost:9001/__services/path/to/service
For example, the service implemented in “tests-intern/services/request/xhr.service.js” can be accessed at:
http://localhost:9001/__services/request/xhr

Code coverage

Intern can generate code coverage information that reports what percentage of lines of source code are covered by the tests. Istanbul (included in Intern’s dependencies) is used to generate coverage information. You should use this information as part of your efforts to determine if the converted tests sufficiently cover the code included in a particular module. As noted, the Grunt task provided will allow you to view code coverage analysis results automatically.

Resources

Dojo development efforts are discussed on the dojo-interest mailing list, so if you have further questions related to writing tests for Dojo you can ask there. General Intern usage issues and help are available on Stack Overflow.

Get involved

We appreciate any time and effort you may contribute towards migrating all of Dojo’s test to use Intern. To volunteer to convert tests, make sure you have a Dojo Foundation CLA on file, and then volunteer to convert tests for Dojo, Dijit, and/or DojoX. This will allow us to more efficiently maintain and release updates for many versions of Dojo without introducing regressions.

Intern webcast

Want to learn more about getting started with Intern? Join us for a free introduction to Intern on March 20, 2014. Register today to attend our Intern webcast.

Comments