The long and winding road to Dojo 2

By on August 24, 2016 10:46 am

Recently on GitHub, someone accused Dojo 2 of being vapourware. This opinion came from a position of misinformation. I was glad the individual then engaged with the Dojo 2 project to understand where we are today. We are making swift progress and a beta is on the horizon. It has taken Dojo 2 a long time to get here and to really solidify our vision. We first started brainstorming about plans for 2.0 almost 5 years ago! Around a year ago we solidified our plans and have been unwavering in moving down that path.

I am really excited about what we have accomplished and what is to come. dojo/meta provides a live table of all our package statuses:

Table of Dojo 2 Package Status

A view of the current status of Dojo 2

The ramp up

One of the biggest challenges for Dojo 1 users was getting up to speed. Part of that problem was the world in which Dojo 1 found itself, where things like “toolchains” were nascent concepts and there wasn’t as wide and rich of an ecosystem of tooling to help build solid code. In the early Dojo 1 days, a developer would only need to download an archive, setup a web server, read three tutorials and maybe spend a couple days iterating to get started with Dojo.

With Dojo 2, we have a getting started goal: “if an average JavaScript developer, using a single tutorial, cannot get a simple app up and running in 15 minutes, then we have failed.” Providing this level of simplicity is very complex and the effort is largely hidden from the end user.

We see the biggest part of providing simplicity as the creation of an out-of-the-box toolchain, which will be focused around dojo/cli. While still in development, we see this as a critical tool to be able to do things like:

$ dojo create myFirstApp

which can setup a new Dojo 2 project for you and provide you with sensible defaults and an efficient toolchain experience.

At the same time, we know our opinion of what a toolchain or workflow is might be problematic in certain runtimes, so we strive to ensure that there is no lock-in to any particular workflow. We are trying out many different ways of packaging Dojo 2 so it will work in as many environments and workflows as possible. We want to be opinionated where it is helpful in discouraging anti-patterns, yet flexible where developers need us to be. We still have a ways to go, but we are headed in the right direction to make ensure Dojo 2 has awesome usability.

Dog food

Dojo 1 grew organically, tempered by the needs of the projects using it. A challenge that met Dojo 2 was that it needed to meet a set of theoretical requirements, leveraging modern technologies and best practices, while at the same time trying to be something that was familiar to those who had worked with Dojo 1.

Unlike Dojo 1, with Dojo 2, we have a strong concept of what an application is. We have enough components built for Dojo 2 that it was time to figure out how we could effectively build an application on top of Dojo 2.

So we started with the typical starting point, the TodoMVC reference application. This effort allowed us to better evaluate what we were missing, what worked well and what was just painful. The source code for the application is available on dojo/examples and you may also see the initial Dojo 2 ToDoMVC application in action.

A screenshot of TodoMVC written in Dojo 2

Example of the TodoMVC reference application written in Dojo 2

This underlined for us how much we had accomplished, but also that we had a few gaps: we need to solidify our build tooling and that we need ways to efficiently transform data from one shape to another.

Build it!

One of the big things we know we need to tackle is to make it easy to provide a production-ready version of a project.

From the early days of Dojo 1, the world has changed. We have seen the rise of Node.js, but it uses a module loader and format that aren’t very well suited for web. We have seen the rise of AMD (and the conversion of Dojo 1.7+ to it), though it is more popular as a compilation format than for its authoring syntax.

ES2015 added a proper module specification (ESM) to JavaScript, but it lacks a loader specification which is still a work in progress. The SystemJS has taken the lead on iterating through the challenges of loading modules, but there are frustratingly no production-ready native loaders and the browser vendors are finding that having parity with SystemJS may be challenging. To try to make this easier, we have seen a proliferation of build tools, including Browserify, WebPack and Rollup.

All of these have benefits (and limitations) and we know it is one of the most confusing parts of JavaScript at the moment. We have to provide a “just add code” experience but also provide flexibility and information on how to integrate it into other toolchains. Modularity is one of the key elements to robust JavaScript application architecture. We have been doing a lot of research and prototyping to find the right solution to this challenge.

For each Dojo package, we plan to have two versions available, a ready to go version where the modules will be in a UMD format, designed for straightforward development or using server-side under Node.js without additional configuration. The other format will be what we are currently calling the über format which will include the TypeScript sources, UMD formatted modules and ESM modules. These are designed to be able to integrate into a larger project, but will likely require additional toolchain configuration and packaging before the package is usable in a project.

You had one job!

One of the big challenges that Dojo 1 ran into was that its extreme flexibility made it difficult for larger scale applications to follow a consistent architecture. There were a few attempts over the years to add a Dojo 1.x application framework, but Dojo 1.x wasn’t coherently designed around the specific concept of a consistent and unifying application architecture.

Again, like a lot of Dojo 2, we want to balance having a very strong opinion of the “Dojo way”, but also not force users of Dojo 2 to accept all of our paradigms. We need an application framework that would fit the sort of enterprise workflow patterns that we have seen and helped evolve over the years. We typically see that business logic, user interface design, branding, service orchestration and final application composition tend to be separate roles within large organizations. Often there are developers engaged specifically in rapidly compositing different interface widgets with business logic and back-end data services to meet business requirements. The ability to do this rapidly and not have to worry about all the details of the implementation is critical to a successful framework.

This is why we have been doing a lot of work on dojo/app which will allow developers to quickly describe an application and orchestrate the components without having to deep dive into specific implementation details. There will be three ways of describing a Dojo 2 application:

  • Declarative Syntax – For those who are familiar with Dojo 1’s parser, the declarative syntax will allow the developer to describe a full application in HTML.
  • Simple Object Syntax – This syntax allows you to describe an application in a simple object format which is easily serialisable into JSON. This will open up significant possibilities for applications to be generated server-side in another framework and realised in an environment.
  • Functional Syntax – This will allow a developer to fully leverage the benefits of TypeScript by providing a functional interface for dealing with the assembly of an application.

We still have a ways to go, but we have been doing the development iteratively and we strive to deliver a stable API for the Dojo 2 beta.

Data, data and data

Data is one of the fundamental tenants of Dojo 2. We know that many of the applications that we would expect to build with Dojo 2 will be data rich, representing huge amounts of data persisted in back-end business systems with a window on that data being present in the application.

So outside of continuing to build upon the paradigms we had in Dojo 1 data object stores and the SitePen project dstore, we know that we need some powerful APIs that allow data to be queried and transformed to be consumed in an application. We also realise that doing all that work in the browser is not a viable option in every case, but having an API that doesn’t require a developer to recode when moving from one type of store to another is critical.

So we have started working on a robust data querying and transformation API that will allow the building of implementations that work client-side or offload to various server-side solutions. We are in the process of figuring out if this will simply be baked into dojo/stores or if it will be a standalone package.

But I need to see my data!

Of course once we provide APIs to access data, the next big thing is the ability to visualise that data. We have started work on a data visualisation library. We wanted something that would align to the patterns we had established for Dojo 2 widgets; widgets are reactive UI elements which take an observable set of data and render to a virtual DOM.

Here is an early, un-themed example of the work:

A screenshot of charts from an early example of Dojo 2 data visualization

We really have to give credit to Mark Wubben for making such rapid progress on this. Once we solidify a few things, we will have an alpha available in the very near future.

Hindsight

In hindsight, this is the perfect time for Dojo 2 to be getting close to completion. The biggest challenge with a version 2 release is the desire to make everything perfect and solve everything you could not solve in version 1.

If we had tried to finish Dojo 2 just two years ago, we would be looking at a complete rewrite to leverage the modern features of ES6 and TypeScript. TypeScript 2 has many features that are critical for building a powerful framework for creating modern applications.

That said, had we known it would take this long, we probably would have greatly refactoring the existing 1.x codebase, called it 2.0, and then the current efforts would have been Dojo 3!

The slog

With all this progress, you might think there isn’t much left to do. There are still a few key things we need to finish. In particular, we still have a lot of work to get right around internationalisation, localisation and accessibility. These are key functional features that Dojo has solved successfully in the past, that are critical to the enterprise applications we envision people building with Dojo 2, and that we want to get right to make them easier to use with Dojo 2.

While the TodoMVC was very informative to us, it wasn’t feature-rich enough to really flesh out the functionality we need. We have started work on our next example application, which will be significantly more complex. It will be data rich and leverage the complexity of the single page app. Parts of the application will come in and out of existence where the application path will keep in sync with the application view. Invariably there will be even more lessons learned that we will need to work on as we refine Dojo towards its 2.0 release!

But overall I think we are headed the right direction and have good velocity in creating a powerful approach for building modern enterprise web applications. I am really looking forward to what the next few months hold!


Learning more

Support Logo

Get help from SitePen Support, our fast and efficient solutions to JavaScript development problems of any size.

Workshops Logo

SitePen workshops are a fun, hands-on way to keep up with JavaScript development and testing best practices. Register for an online workshop, today!

Let's Talk! Logo

Let’s talk about how we can help your organization improve their approach to automated testing.

Contact Us Logo

Have a question? We’re here to help! Get in touch and let’s see how we can work together.

Comments

  • Corey Alix

    Regarding Simple Object Syntax, it would be helpful to have a configuration tool for constructing these SOS documents and then have the app framework that can deep-merge them into a complete application. Each sos is like middleware but maybe “innerware” representing modules, widgets, configuration overrides or any extension to an existing sos:

    “` App.build(sos1, sos2, sos3).then(app => app.run());

    So sos1 might be a base application, sos2 puts specific controls into the sos1 shell and sos3 puts some concrete configurations into the models associated with sos2 (for instance). That’s something I’d like to see.

  • Pong

    I hope that, the dojo document page should create a user contribute notes just like the php website
    http://php.net/manual/en/function.substr.php

    development can share their experience directly in the function manual rather than ask question in mailing list. Mailing list is not a convenience way for share experience

  • We had this early on with Dojo 1.x, and it was eventually overrun by spammers so we moved docs to GitHub. We’ll see what we can do, but a main consideration is a system that does a decent job against relentless spammers.

  • MUHAMMAD SHAHZAD

    Awesome, good to hear about the progress, good luck guys, i can’t wait to get my hands on dojo2.

    With that in mind, can you describe how hard it will be in terms of migration, like if we want to migrate dojo 1.9 app to dojo 2.0. Is that even feasible?

  • The short answer I would give is that ES6 happened, which fundamentally changed the underlying landscape. So we really needed to step back and rethink how we wanted to move forward. That and after years of enterprises resisting dropping IE6-8, suddenly everyone was fine dropping those old browsers, which meant that we had a larger than expected amount of cruft to leave behind.

  • We currently have a pretty easy strategy for including 1.x widgets inside Dojo 2 apps. Beyond that, we’re not sure yet. Migration strategies are often something you work on after you know what you’re migrating to. There’s a pretty significant amount of change for the better, so a clean rewrite may make more sense, but we just don’t know yet if we’ll have a better option than that. We will continue to support 1.x users for a long time as well.

  • Interesting comment/question @coreyalix. In your example, is each sos an orthogonal concern, or a vertical slide of functionality? With the former, you could use dojo/cli https://github.com/dojo/cli/issues/8 to compose an app, e.g. dojo app build “sos1 sos2 sos3” . Is this anything more than order-dependent module fetching? Perhaps you want to take a look through the plans for dojo-cli and give us more feedback there?

  • Corey Alix

    You might have this covered with app.loadDefinition(). That is very powerful to temporarily extend the app (e.g. via a tool selection) by configuring the tool (actions, stores, widgets, etc), ‘realizing’ the tool and finally cleaning up with handle returned by loadDefinition. I should have looked first!

  • W.S. Hager

    Agreed. ES6 appeared. So did React. However the ‘cruft’ wasn’t just old browsers, but rather the legacy Dojo put on itself through the multiple inheritance OOP approach. There were even proposals to draw that legacy into Typescript, but it stranded there. Compose already existed, and there were other, more native, alternatives. So ES6 could have happened regardless. And as for React… HTML is still just XML, which has an impressive tool chain (let’s call it virtual DOM). Dojo could’ve just cleaned up 1.x and ignore the rest. Not blaming anyone, but this kind of consensus scares me.

  • I don’t think there’s consensus for the sake of consensus by any means, but several months of work to evaluate options.

    Taking a step back, dgrid is in essence a special case of virtual DOM.

    With regards to TypeScript, in the early days, we ran into a wall with inheritance that could be efficiently represented in TS and normal JS. dojo/compose is the result of months of work to make that happen (along with work to get TS into a place it needed to be to help make that happen).

    I do agree in hindsight that we should have cleaned up 1.x, called it 2.0, and what we’re doing now would be 3.0.

  • Steve Kerr

    Exciting news. I expect down the track we will do a full rewrite using Dojo 2.0 + React Native. Looking Forward to it.

  • Macrow Willson

    i just wish dojo-2 doesnt become a heavy software with too many dependencies as it looks now already. using dojo-1 now for a almost a decade, it was a great learning experience and it was relative easy to replace the dojo parts with our own implementation. it paid well out to do so, we’re less impacted by updates and it keeps things easy to maintain, by a single person! i don’t see how one can do this with dojo2. it seems really complicated to add changes to the core and keep updated the same time…. anyway, i am happy to see that all the good skills and experiences go into something new.

  • As we’re iterating through development of Dojo 2, it definitely seems more difficult, but this would have been the case as well if you tried to use Dojo between 0.4 and 1.0 as we were iterating extensively.

    One of our goals is that our users would not be exposed to the challenge of managing versions and changes across 20+ packages (unless they just want to use a few of the packages). So it should be easy to stay current and upgrade and extend, with more atomic releases.

    It should be very easy to extend/replace features as well (either via dojo/compose or ES6 modules or UMD modules). We still need to write tutorials and documentation and provide more examples, which will explain how these patterns will work for Dojo 2. But it wouldn’t be terribly useful to create those docs until we feel that the APIs are stable.

  • Macrow Willson

    thanks Dylan, in your case it may worth to give a “docs first” a shot. i experienced surprising good effects with larger code base(s)

  • To be fair, we’re doing a lot more than before in this regard. We do have a fair amount of that on the package level, e.g. https://github.com/dojo/app/ or https://github.com/dojo/compose . And even before that with all of the documents linked from http://dojotoolkit.org/community/roadmap/ . Where it quickly falls behind though is API iteration as well as docs on how to use everything together. What might be most immediately interesting to you is the first alpha of dojo-cli, https://github.com/dojo/cli , along with docs https://github.com/dojo/cli/issues/8 and https://github.com/dojo/cli/issues/9 . This is basically for the command line tool to help you get setup with Dojo 2!

  • Macrow Willson

    wow, didn’t noticed the progress, thanks Dylan.