The long and winding road to Dojo 2

By on August 24, 2016 10:46 am

Recently on GitHub, someone accused Dojo 2 of being vapourware. This opinion came from a position of misinformation. I was glad the individual then engaged with the Dojo 2 project to understand where we are today. We are making swift progress and a beta is on the horizon. It has taken Dojo 2 a long time to get here and to really solidify our vision. We first started brainstorming about plans for 2.0 almost 5 years ago! Around a year ago we solidified our plans and have been unwavering in moving down that path.

I am really excited about what we have accomplished and what is to come. dojo/meta provides a live table of all our package statuses:

Table of Dojo 2 Package Status

A view of the current status of Dojo 2

The ramp up

One of the biggest challenges for Dojo 1 users was getting up to speed. Part of that problem was the world in which Dojo 1 found itself, where things like “toolchains” were nascent concepts and there wasn’t as wide and rich of an ecosystem of tooling to help build solid code. In the early Dojo 1 days, a developer would only need to download an archive, setup a web server, read three tutorials and maybe spend a couple days iterating to get started with Dojo.

With Dojo 2, we have a getting started goal: “if an average JavaScript developer, using a single tutorial, cannot get a simple app up and running in 15 minutes, then we have failed.” Providing this level of simplicity is very complex and the effort is largely hidden from the end user.

We see the biggest part of providing simplicity as the creation of an out-of-the-box toolchain, which will be focused around dojo/cli. While still in development, we see this as a critical tool to be able to do things like:

$ dojo create myFirstApp

which can setup a new Dojo 2 project for you and provide you with sensible defaults and an efficient toolchain experience.

At the same time, we know our opinion of what a toolchain or workflow is might be problematic in certain runtimes, so we strive to ensure that there is no lock-in to any particular workflow. We are trying out many different ways of packaging Dojo 2 so it will work in as many environments and workflows as possible. We want to be opinionated where it is helpful in discouraging anti-patterns, yet flexible where developers need us to be. We still have a ways to go, but we are headed in the right direction to make ensure Dojo 2 has awesome usability.

Dog food

Dojo 1 grew organically, tempered by the needs of the projects using it. A challenge that met Dojo 2 was that it needed to meet a set of theoretical requirements, leveraging modern technologies and best practices, while at the same time trying to be something that was familiar to those who had worked with Dojo 1.

Unlike Dojo 1, with Dojo 2, we have a strong concept of what an application is. We have enough components built for Dojo 2 that it was time to figure out how we could effectively build an application on top of Dojo 2.

So we started with the typical starting point, the TodoMVC reference application. This effort allowed us to better evaluate what we were missing, what worked well and what was just painful. The source code for the application is available on dojo/examples and you may also see the initial Dojo 2 ToDoMVC application in action.

A screenshot of TodoMVC written in Dojo 2

Example of the TodoMVC reference application written in Dojo 2

This underlined for us how much we had accomplished, but also that we had a few gaps: we need to solidify our build tooling and that we need ways to efficiently transform data from one shape to another.

Build it!

One of the big things we know we need to tackle is to make it easy to provide a production-ready version of a project.

From the early days of Dojo 1, the world has changed. We have seen the rise of Node.js, but it uses a module loader and format that aren’t very well suited for web. We have seen the rise of AMD (and the conversion of Dojo 1.7+ to it), though it is more popular as a compilation format than for its authoring syntax.

ES2015 added a proper module specification (ESM) to JavaScript, but it lacks a loader specification which is still a work in progress. The SystemJS has taken the lead on iterating through the challenges of loading modules, but there are frustratingly no production-ready native loaders and the browser vendors are finding that having parity with SystemJS may be challenging. To try to make this easier, we have seen a proliferation of build tools, including Browserify, WebPack and Rollup.

All of these have benefits (and limitations) and we know it is one of the most confusing parts of JavaScript at the moment. We have to provide a “just add code” experience but also provide flexibility and information on how to integrate it into other toolchains. Modularity is one of the key elements to robust JavaScript application architecture. We have been doing a lot of research and prototyping to find the right solution to this challenge.

For each Dojo package, we plan to have two versions available, a ready to go version where the modules will be in a UMD format, designed for straightforward development or using server-side under Node.js without additional configuration. The other format will be what we are currently calling the über format which will include the TypeScript sources, UMD formatted modules and ESM modules. These are designed to be able to integrate into a larger project, but will likely require additional toolchain configuration and packaging before the package is usable in a project.

You had one job!

One of the big challenges that Dojo 1 ran into was that its extreme flexibility made it difficult for larger scale applications to follow a consistent architecture. There were a few attempts over the years to add a Dojo 1.x application framework, but Dojo 1.x wasn’t coherently designed around the specific concept of a consistent and unifying application architecture.

Again, like a lot of Dojo 2, we want to balance having a very strong opinion of the “Dojo way”, but also not force users of Dojo 2 to accept all of our paradigms. We need an application framework that would fit the sort of enterprise workflow patterns that we have seen and helped evolve over the years. We typically see that business logic, user interface design, branding, service orchestration and final application composition tend to be separate roles within large organizations. Often there are developers engaged specifically in rapidly compositing different interface widgets with business logic and back-end data services to meet business requirements. The ability to do this rapidly and not have to worry about all the details of the implementation is critical to a successful framework.

This is why we have been doing a lot of work on dojo/app which will allow developers to quickly describe an application and orchestrate the components without having to deep dive into specific implementation details. There will be three ways of describing a Dojo 2 application:

  • Declarative Syntax – For those who are familiar with Dojo 1’s parser, the declarative syntax will allow the developer to describe a full application in HTML.
  • Simple Object Syntax – This syntax allows you to describe an application in a simple object format which is easily serialisable into JSON. This will open up significant possibilities for applications to be generated server-side in another framework and realised in an environment.
  • Functional Syntax – This will allow a developer to fully leverage the benefits of TypeScript by providing a functional interface for dealing with the assembly of an application.

We still have a ways to go, but we have been doing the development iteratively and we strive to deliver a stable API for the Dojo 2 beta.

Data, data and data

Data is one of the fundamental tenants of Dojo 2. We know that many of the applications that we would expect to build with Dojo 2 will be data rich, representing huge amounts of data persisted in back-end business systems with a window on that data being present in the application.

So outside of continuing to build upon the paradigms we had in Dojo 1 data object stores and the SitePen project dstore, we know that we need some powerful APIs that allow data to be queried and transformed to be consumed in an application. We also realise that doing all that work in the browser is not a viable option in every case, but having an API that doesn’t require a developer to recode when moving from one type of store to another is critical.

So we have started working on a robust data querying and transformation API that will allow the building of implementations that work client-side or offload to various server-side solutions. We are in the process of figuring out if this will simply be baked into dojo/stores or if it will be a standalone package.

But I need to see my data!

Of course once we provide APIs to access data, the next big thing is the ability to visualise that data. We have started work on a data visualisation library. We wanted something that would align to the patterns we had established for Dojo 2 widgets; widgets are reactive UI elements which take an observable set of data and render to a virtual DOM.

Here is an early, un-themed example of the work:

A screenshot of charts from an early example of Dojo 2 data visualization

We really have to give credit to Mark Wubben for making such rapid progress on this. Once we solidify a few things, we will have an alpha available in the very near future.

Hindsight

In hindsight, this is the perfect time for Dojo 2 to be getting close to completion. The biggest challenge with a version 2 release is the desire to make everything perfect and solve everything you could not solve in version 1.

If we had tried to finish Dojo 2 just two years ago, we would be looking at a complete rewrite to leverage the modern features of ES6 and TypeScript. TypeScript 2 has many features that are critical for building a powerful framework for creating modern applications.

That said, had we known it would take this long, we probably would have greatly refactoring the existing 1.x codebase, called it 2.0, and then the current efforts would have been Dojo 3!

The slog

With all this progress, you might think there isn’t much left to do. There are still a few key things we need to finish. In particular, we still have a lot of work to get right around internationalisation, localisation and accessibility. These are key functional features that Dojo has solved successfully in the past, that are critical to the enterprise applications we envision people building with Dojo 2, and that we want to get right to make them easier to use with Dojo 2.

While the TodoMVC was very informative to us, it wasn’t feature-rich enough to really flesh out the functionality we need. We have started work on our next example application, which will be significantly more complex. It will be data rich and leverage the complexity of the single page app. Parts of the application will come in and out of existence where the application path will keep in sync with the application view. Invariably there will be even more lessons learned that we will need to work on as we refine Dojo towards its 2.0 release!

But overall I think we are headed the right direction and have good velocity in creating a powerful approach for building modern enterprise web applications. I am really looking forward to what the next few months hold!


Learning more

Support Logo

Get help from SitePen Support, our fast and efficient solutions to JavaScript development problems of any size.

Workshops Logo

SitePen workshops are a fun, hands-on way to keep up with JavaScript development and testing best practices. Register for an online workshop, today!

Let's Talk! Logo

Let’s talk about how we can help your organization improve their approach to automated testing.

Contact Us Logo

Have a question? We’re here to help! Get in touch and let’s see how we can work together.