Ember's conventions for project layout and file naming are central to the
developer experience. It's crucial that we get both the technical and
ergonomic details right. I wanted to provide an update about Module
Unification and our plans for the file structure in Ember Octane.
In short, we do not plan to ship Module Unification in Octane. Instead,
Octane will ship with today's file system layout, with one change: support
for nested components in invocation syntax
Because Octane apps will continue with today's file system layout, we want to
address the largest barrier to adoption today: components
nested inside other directories.
For example, if you have a component located at
app/components/icons/download-icon.js (i.e., nested inside an icons
directory), you can invoke it with curly invocation syntax like this:
{{icons/download-icon}}
However, it's not possible to invoke the same nested component with angle
bracket syntax without resorting to clunky workarounds.
As proposed in the Nested Invocations in Angle Bracket Syntax
RFC, we plan to address this by adding support for
nested components via the :: separator.
With this proposal, the component described above could be invoked with angle
bracket syntax like this:
Because this is a small change, it can be implemented quickly without
requiring us to delay the Ember Octane release.
Status of Module Unification
Given that the Module Unification RFC was merged in late 2016, and we talked
about shipping Module Unification in the 2018 Roadmap RFC,
it's fair to ask what happened and why we're making this decision now.
Heads up: this post gets long and detailed, so if you only care about the
plan going forward, you can safely skip the rest.
In the spirit of transparency and overcommunication, though, I wanted to
share a little bit of the history and evolution of MU from my perspective.
Let's call the file system layout that Ember apps use today the "classic"
layout. While this structure has served us well, it also has several
shortcomings.
In the classic layout, files are grouped by type, not by name. Sometimes,
this means that closely related files (like a component and its template) are
separated from each other, and navigating between them can be frustrating.
Ideally, related files would be close to each other in the file system. For
example, you may want an Ember Data model and its associated serializer to be
co-located in the same directory.
Early on, Ember CLI implemented an experimental "pods" layout that grouped
files by name rather than by kind. For example, a User model and its
serializer would be grouped together in app/user/, as model.js and
serializer.js respectively.
Feedback from the community was that pods felt more productive than the
classic layout. However, pods had several problems that needed to be
addressed before it could be enabled by default.
The effort to address the design flaws of pods led to the Module Unification
RFC, a ground-up rethink of how the file system should work in Ember
apps. Importantly, this design grappled with overhauling Ember's resolver and
dependency injection systems as well, which was just as important as
shuffling around where particular files went on disk.
The MU RFC was merged at the end of 2016, but new Ember apps are still
generated with the classic layout. So, why can't we flip the switch on
enabling Module Unification-style layout by default today?
Remember that MU described not just a new file system but a significant
overhaul of Ember's resolver. Implementing MU was a huge initiative that,
while often hidden, touched nearly every part of the framework. Despite this
large scope, and thanks to the incredible amount of time and energy our
community devoted to the work, MU implementation has made great progress, and
nearly everything needed to make Module Unification work has landed in Ember.
Nearly everything. When we merged the Roadmap RFC last year, there was one
last major piece of MU that still needed to be designed: component
namespacing, or how to refer to components that come from other addons in
templates.
While we had a plan, a shift in the JavaScript ecosystem sent us back to the
drawing board. And the harder we worked on solving this last piece, the more
we realized that fundamental aspects of the overall design needed to be
rethought.
Namespaces and Scoped Packages
Besides files being spread out, another problem with the classic layout is
that everything goes into one big global pool of names.
So, for example, Ember Data defines a service called store that you can
inject into components and services. If you install another addon that also
has a service called store, there's no easy way to use both.
Similarly, if your app has a component called small-button and you install
an addon that also has a component called small-button, you have no way to
tell Ember which one you mean when you type {{small-button}} in a template.
One of the key benefits of the MU design is that names are no longer global,
but namespaced to the package where they come from. So, for example, if an
npm package called ember-ui-library contains a component called
small-button, the MU RFC proposed referring to it in your template like
this:
{{#ember-ui-library::small-button}}
...
{{/ember-ui-library::small-button}}
Around the same time we were designing MU, however, npm announced support
for scoped package names. Prior to this change, npm package
names were limited to alphanumeric characters and dashes. Now, though,
packages could start with an @ and contain a /, like
@glimmer/component.
Component invocations with scoped packages blew past the verbosity limit. We
simply could not bring ourselves to accept a syntax that commonly produced
code like this:
{{#@my-company/ember-ui-library::small-button}}
...
{{/@my-company/ember-ui-library::small-button}}
We also had concerns that this was confusing to scan visually, considering
that @ already means a component argument and / already means a closing
tag.
While Ember had been using it for years, around this time JavaScript module
syntax (import Post from './post') started gaining significant traction in
the wider ecosystem, along with tools like webpack that could use these
modules to perform code splitting.
After scoped npm packages scuttled our original plan for namespaced
components, we went back to the drawing board, and something similar to
JavaScript's import seemed like a promising solution. However, we
immediately hit some challenges while exploring this idea.
The concept of a "component" in Ember isn't a singular thing, but the union
of a template and a component class. With the component and template in
separate files, it isn't clear which one you're supposed to import. Because
of this, we wanted something where you would import a component by name
(my-component), not a specific file like in JavaScript.
But despite the differences, the overall concept of importing modules was
similar. We wanted to find a syntax that would give users who already knew
JavaScript an intuition about the similarities, while not making it look so
similar that people would be misled into thinking it was literally the same
thing.
We tried to find a balance between these two constraints with a syntax that
looked like this:
{{use component-name from 'package-name'}}
We hoped that it looked similar enough to JavaScript's import syntax to
give you a clue about what it was doing, but by adopting the use keyword
instead of import, signal that this was not exactly the same as JavaScript
modules.
Matthew Beale poured significant time into capturing all of these conflicting
constraints in the Module Unification Packages RFC, but, even after
months of discussion, we couldn't come to consensus and the RFC was never
merged.
Despite everyone agreeing this was an urgent problem, we couldn't convince
ourselves that having different module systems for JavaScript and templates
was a viable solution. Unfortunately, there wasn't an obvious alternative
plan, and not having an answer meant MU was indefinitely blocked until we
could figure out this last piece of the puzzle.
We felt stuck.
Real-World Feedback
In the meantime, enough of Module Unification was shipping behind feature
flags (and in Glimmer.js) that we were able to get feedback from early
adopters. While overall people really liked the new file system and really
appreciated not dealing with frustrating name collisions, something felt
off.
One common theme in the feedback was that MU felt too rigid and frequently
got in the way of simple tasks. To understand why, it's important to
understand that MU is about more than a file system. MU is really a system
for controlling scope.
For example, a feature of MU is the ability to have private components that
don't leak into the rest of the app. If we have a component called
list-paginator and it has a child component called paginator-control, MU
allows us to organize them like this:
src
├── ui
│ ├── components
│ │ └── list-paginator
│ │ ├── paginator-control
│ │ │ ├── component.js
│ │ │ └── template.hbs
│ │ ├── component.js
│ │ └── template.js
In this example, the list-paginator template can invoke
{{paginator-control}} to render its child component. However, if you try to
invoke {{paginator-control}} from any template outside the list-paginator
directory, you'll get an error. In other words, paginator-control is
local to list-paginator.
MU, then, is about scope, and controlling who has access to what. Where a
module lives in the MU file system determines what it can see to and who
else can see it.
This is a clever idea that eliminates a lot of boilerplate. If you have to
organize your files anyway, and if you want to group related things together
anyway, it makes sense to try to create a single system that solves
organization and scoping at the same time.
In practice, though, we ran into a few challenges:
- This idea is not common in the JavaScript ecosystem, so the file system
controlling scope isn't intuitive for new learners. They also have to
memorize these naming rules, which are implicit and get quite complex. - Similarly, ecosystem tools don't understand MU. We have to build custom
integrations to get things like "Go to Definition" to work in IDEs or code
splitting to work in webpack, that other libraries get for free. - JavaScript files in Ember use module syntax, which doesn't go through the
MU system, adding to the confusion. Having one system in a component's
JavaScript and another in its template is not ideal. - When file names and locations are so meaningful, it can be frustrating
if you want to create a file that isn't part of the MU world. Tasks that
are normally trivial, such as extracting utility functions into a separate
file or grouping related files together in a directory, can easily turn
into a battle where your build starts erroring because you're not playing
by the MU rules.
A Personal Anecdote
Personally, this last one was what really caused me to step back and
re-evaluate our plan for MU. It happened during a project at work where we
were using both Glimmer.js (with Module Unification) and Preact.
As the number of components grew, a co-worker created a directory called
icons in the Preact app to hold all of the components for rendering
different SVG icons. I'm sure it didn't take more than a few minutes to
create the directory, drag the appropriate component files in, and update the
paths everywhere those components were imported. (In fact, VS Code probably
updated the import paths automatically.)
When we tried to do the same thing in the Glimmer app, it was a much
different experience that turned into an hours-long slog. And despite all the
great things it does do, MU doesn't really have a way to do this kind of
lightweight grouping.
We could have found a workaround. We could have extracted the SVG icon
components into a separate package, or created a higher-order component that
wrapped all of the child icon components. But it seemed ridiculous to burn so
much time looking for a "workaround" to perform a task that felt like it
should have been (and was, in Preact) trivial.
I knew, intellectually, the benefits of MU. I knew how carefully it was
designed to enforce structure and consistency as your application grew and
had different engineers of different experience levels working on it. (Indeed,
by the end of the project, I found the Glimmer app much easier to navigate,
while the Preact app had several inconsistently-followed conventions.)
But I never forgot how viscerally bad it felt to have my co-workers fight so
hard to do something that felt like it should be so easy.
So this was the status quo last year. We were all incredibly frustrated that
we couldn't make progress on the scoped package problem, but I was even more
overwhelmed trying to figure out what, if anything, to do about the negative
experiences my co-workers had had when using MU.
Discovering problems in a design you've been working on for so long is
painful, especially in cases like this where the majority of the work was
already complete, and we thought we were so close to the finish line.
Programming language and API design is really hard. Sometimes I read old RFCs
and marvel at how obvious the solution seems now, in contrast to the weeks,
months, or years I know it took to tease it out from the millions of possible
alternatives.
When you're trying to balance so many competing constraints, sometimes a
small change is all it takes to get you out of a design conundrum you've
struggled with for months. In this case, that change was angle bracket
component invocation.
One thing making JavaScript imports difficult to use with templates was the
constraint that components had to have a dash in their name. Unfortunately,
dashes aren't valid in JavaScript identifiers, so something like import produces a syntax error.
some-component from './some-component'
Angle bracket components, on the other hand, start with a capital letter to
disambiguate themselves from normal HTML tags: instead of
{{my-component}}. Most importantly, there's no dash.
As the Ember community started using angle bracket syntax, early feedback was
very positive. All of a sudden, JavaScript import syntax was back on the
table.
The Path Forward: Template Imports
Speaking for the Ember.js core team, we are trying to get better at updating
the community when plans have changed but the new plan isn't fully locked in
yet. So, consider this one of those situations.
We know that the exact plan for Module Unification (MU), as described in the
original RFC, will need to change. How it changes is not yet certain,
but we believe that some of the problems we wanted to solve with MU are
better solved with template imports.
With template imports, we intend to make templates play nicely with
JavaScript, so you can use the import feature you already know and love. By
having components and helpers be modules you can import, we can eliminate the
most complex parts of Module Unification so it's easier to learn and use.
We recently posted the SFC & Template Imports RFC,
which describes some of the low-level APIs needed in Ember to make template
imports possible.
Learning from past mistakes, this RFC focuses on the primitives so we can
quickly experiment, get feedback, and iterate on template import proposals in
addons, before stabilizing them in the core framework.
While the Ember.js core team has reached consensus that template imports are
the intended path forward, please note that the current RFC only covers
low-level primitives, not the API that would be used by Ember developers to
author templates.
Here is one example of a very hypothetical template imports syntax:
---
import UserProfile from './user-profile';
import UserIcon from './icons/user';
---
{{this.blog.title}}
The syntax is up in the air and will almost certainly be different from this
example. Regardless, it shows the benefit of template imports clearly: we've
imported two components—UserProfile and UserIcon—just like how we would
refer to any other JavaScript module. This makes it very easy for
everyone—from developer who are new to Ember, to IDEs, and other tooling in
the JavaScript ecosystem—to understand what came from where.
You can even imagine an (again, very hypothetical) single-file component
format that places the template right within the component's class. Here, a
unified imports solution feels especially natural:
// src/ui/components/blog-post.gbs
import Component from '@glimmer/component';
import UserIcon from './icons/user';
export class BlogPost extends Component {
blog = { title: 'Coming soon in Octane', authorId: '1234' };
{{this.blog.title}}
}
Again, the exact syntax is up in the air and will almost certainly be
different from this example. The benefit of exposing the low-level primitives
first is that we can try out multiple competing designs relatively easily
before comitting. And if you don't like what we eventually decide on, you can
build an alternative that is just as stable as the default implementation.
But note that template imports are not a replacement for MU. They don't help
with things like better isolation of an addon's services, or a more intuitive
file system layout. Instead, we hope that template imports will better solve
one aspect of MU, so the overall design can be simplified and its benefits
can shine through more clearly.
Template imports also give us a good opportunity to try to address the
ergonomic problems people reported when trying MU.
Without template imports, we had to rely on MU to resolve component names,
meaning the files in the src/ui/components directory had to follow strict
rules. But with template imports, users can just tell us which module on
disk they want. We can skip resolving components through MU altogether, and
let Ember users organize their component files however they want.
As frustrating as it was at the time, the inability to make progress on MU
may have been a blessing in disguise. It gave us time to implement angle
bracket syntax for components, which allowed template imports to seem
feasible again—which means we now have a solution that seems to address both
the scoped package problem and the ergonomics problem. Even better, template
imports make things like treeshaking and code-splitting in
Embroider much easier.
I believe the dead-end we found ourselves in was a sign from the universe
that something better was just around the corner. Time will tell, but my
hunch is that template imports solve these important problems much more
elegantly than what we had before. This process also pushed us to explore
single-file components, which I think will be a surprisingly big productivity
improvement for Ember developers.
While we're excited about template imports, we want to keep our promise to
finish what we started. We are prioritizing Ember Octane and making sure that
our first edition is a polished, cohesive experience. Only once Octane is out
the door will we turn our attention to new initiatives like template imports.
Hopefully, this post helps provide some context about the state of MU. Of
course, what I've written here is my personal, imperfect recollection of
events, simplified for clarity. The reality of technical design is messy and
feels a lot more like going around in circles than the tidy sequence I've
presented here.
I will also mention that, as a project, I think we've dramatically improved
how we design, implement, validate, and iterate on features since we
originally started the Module Unification RFC. The MU RFC is the last of the
proposals from the "mega-RFC" era, where we had a tendency to do a ton of
upfront design and implementation before we had any feedback from real users.
Nowadays, I think we're a lot better about making sure RFCs are relatively
small and focused on doing one thing. We also tend to prioritize exposing
hooks or other primitives that let us test out new ideas in addons. This
allows us to improve designs based on early feedback, without the strict
stability requirements that come with landing something in the framework
proper.
This has worked well for things like ember-decorators and Glimmer
components, where real world feedback and the ability to make breaking
changes based on that feedback was critical. I'm hopeful that a similar
strategy for template imports will be just as successful.
My sincere thanks to everyone who has worked so hard on MU and related
efforts. I'm proud to be part of a community that refuses to charge ahead
with something we know isn't right. Ember's longevity is, at least in part,
explained by our willingness to make these kinds of hard decisions.
I'm so excited about Ember, our roadmap, and the newfound energy in our
community. In 2019, a thriving Ember is more important for the web than ever.
Thank you for being a part of our community, and I hope to see you at
EmberConf in a few weeks. It's gonna be a good one.