This was a very large amount of work, and we learned a number of interesting things along the way that should be useful to you in writing your own tests.
Limits for large projects
The open source continuous integration and cloud testing stack of popular choice (travis-ci + SauceLabs) has relatively low limits (50 minutes before a job times out), and the virtual machines provided by SauceLabs appear to be fairly underpowered, resulting in tests taking a fairly long time to run, and eventually failing as the time limit is reached. This is a problem as it means we cannot simply hook into these services to quickly validate each potential pull request against our test suite. We’re still working on a possible solution.
One of the main benefits provided by Dojo is a collection of packages that are tested to work with each other, and are generally consistent in their approach.
In our Intern config for Dojo, we specify various package locations and maps to differentiate between Intern’s version of Dojo that is used internally by Intern, and Dojo’s version of Dojo. We also have test modules created to aggregate over all of our tests. A recent addition in Intern 2.1 is a
skip method to the context object for test methods. Using
this.skip() allows you to clearly see in your tests results which tests were skipped (as opposed to using
dojo/has to conditionally load test modules). Within Dojo, we include skip statements for tests we intentionally added for any tests that have not yet been converted from DOH to Intern. With environment-specific tests, ensure that your test execution process covers all relevant environments and that your reporting process verifies coverage.
Mocking and spies
While in many cases, writing easily testable code makes it easy to mock with Intern, there are cases where having a library like Sinon.JS made it much easier to test our code without modifying it to make it easier to test. For example, see the test for dojo/throttle.
One major point to follow with writing good tests is to make sure that the result of one test is not required for the next test. We found a number of cases within existing Dojo DOH tests where this was not the case, but with our new tests using Intern, we’ve taken a more disciplined approach. For example, the original DOH-based
dojo/on tests were somewhat monolithic which made it difficult to parse the intent of the tests and convert. The converted Intern-based
dojo/on tests are decoupled into smaller, more focused pieces that are easier to read and understand.
Unit vs. functional
Anything that can be run as a unit test should be run as a unit test, and functional tests should be limited to things that must be run as a functional test. In general, unit tests are less complex, less fragile, and easier to create than functional tests. Of course, if you find yourself going through contortions and trickery to create a test (e.g. digging into API internals, generating synthetic events), maybe it should be a functional test instead.
As a result of this conversion, we’ve migrated some functional tests to unit tests wherever possible. An example of something that really must be a functional test: dojo/touch, the API for working with touch events.
The Intern packages Leadfoot, and WebDriver in general, allow you to create incredibly fast UI events (mouse, keyboard, etc.), in some cases much faster than a human can create. In some cases, you may need to slow things down to get them to work (e.g.
dojo/dnd) by using
Command#sleep. Other areas where this may be essential include moving the mouse, creating/destroying DOM nodes, and triggering animations.
We can still use some help to complete this effort in time for Dojo 1.11:
- Convert the remaining tests in Dojo core (Hint: The tests that we currently skip are the ones that are missing, see the testing README for instructions)
- Help convert Dijit and DojoX tests
- Help backport tests to earlier Dojo releases once the core conversion is complete
- Help determine a scalable solution to continuous integration
Let us know if you would like to get involved!
Creating an ideal approach to testing for your applications is a rewarding effort, but requires a significant up front investment to create a process that will work right for your organization. Contact us for a free 30 minute consultation to discuss how we can help.