This post was originally published on mainmatter.com

For anyone who has clicked the link to this blog post but doesn't already know,
Embroider is a new build pipeline that compiles Ember apps into spec-compliant,
modern JavaScript. Before Embroider, it was somewhat difficult to participate in
modern build-tooling optimisations such as code-splitting and tree-shaking.
Embroider allows you to opt-into these behaviours out of the box.
Embroider is of critical importance for the entire Ember ecosystem, yet has been
in development for many years with no clear end in sight. The Embroider
Initiative is spearheaded by Mainmatter with the support of a group of sponsors
to help speed up development on Embroider so that we can make it the default
build system for newly generated Ember apps as soon as possible. This will allow
Ember developers to continue leveraging its advantages while benefiting from a
modern build system. We hope to see Ember apps being built with Vite before the
end of the year.
The Embroider Initiative aims to:

1. Finish Embroider itself by assigning experienced Ember engineers
   full-time to the project. This covers development work on Embroider and its
   dependencies, as well as helping backers setting it up in their repos to
   uncover and fix edge cases.
2. Make Embroider maintainable by decentralizing technical knowledge beyond
   the project's core developer Ed Faulkner and thus
   improving the bus factor.
   Mainmatter makes use of the apprenticeship model to onboard and train new
   developers on the intricate parts of the project.
3. Shift the ecosystem and make Embroider mainstream by making it easier to
   generate Embroider-optimized Ember apps and supporting addon developers make
   their addons compatible with Embroider.

To reach these goals and keep up the momentum, the project needs your
financial support. Please get in touch to learn how to directly
benefit from this investment.
Mainmatter founder and director Marco Otte-Witte recently
talked about the Embroider Initiative and open source funding in general at EmberFest in Madrid.
He detailed how important the Initiative is to the future of Ember itself and to
all companies that have built on Ember and need to secure the investments they
have made in Ember. He also went into detail about the Embroider Initiative in
his
original blog post that announced the initiative.
This blog post will dive into the things our team has been able to achieve up
until now.
Getting Ember to work with Vite
At EmberConf this year Ed Faulkner
announced that we were closing in on a Vite plugin for Embroider.
While that was true at the time, we have learned a lot about the Vite build
process since then and we know more about the steps that are still required to
get the Vite integration working.
The Vite app that Ed demoed at EmberConf was a trivial app that is
a package in the Embroider monorepo
and if you wanted to test it yourself then you could either clone the Embroider
monorepo, or you could clone
this repo which is essentially just
extracting the same test app into an independent repo. It works, and you can
even see the incredible rebuild speeds in action.
The issue with this trivial demo is that it doesn't represent an average Ember
application. There aren't any Ember applications out there that don't have a
single addon installed. While it's not exactly true that the demo doesn't have
any addons installed, it doesn't have any addons that are doing any real
work. And, as it turns out, getting dependencies to work right is the challenge
with the Vite build.
Ed and Mainmatter Senior Engineer Chris Manson have been pairing weekly,
plugging away at the remaining things that are required to fix the Vite build.
We are now optimistic we can get to a point where it's possible to build the
first real-world Ember applications with Vite until the end of the year.
ember-auto-import allowAppImports
While the main focus of the Embroider Initiative was always going to be the
Embroider code base, there are other parts of the ecosystem that will require
some work to bring them more in line with how we want people to build their
apps.
If you're already using Embroider, you will know that a lot of the work to
package your app is done by Webpack. If you're still on a classic build, you may
not be aware that ember-auto-import uses Webpack under the hood to allow you to
seamlessly import from node_modules. This has been a very useful feature but
since the acceptance of the
v2 addon spec RFC
we have noticed that we have a bit of a blindspot in classic builds. Since v2
addons can't influence the build in any way (effectively making them static
packages) addon authors need to add extra installation instructions to detail
how to add a Webpack plugin to their application build config if they still
wanted to influence the build process in any way. This is perfectly legitimate
in Embroider but it does not work for classic apps.
The issue is that ember-auto-import was originally designed to only work with
npm packages, so that means that classic apps couldn't add a Webpack plugin that
would influence the build process for any files controlled by their app, only
for files controlled by npm packages or addons. This has been a
blocker for some addon developers who want to upgrade their addons to the new v2 format
and our solution to this problem has been to
add a new config to ember-auto-import
to allow you to specify parts of your app that should be under its control.
While this work has been done to facilitate v2 addons having the same install
instructions for classic ember-cli builds and Embroider apps, this functionality
could also be considered a way to allow you to opt-in to Embroider on a folder
by folder basis and when your whole app is being controlled by ember-auto-import
(and Webpack) the move to embroider should technically require no changes to the
app.
Progress for Embroider Initiative backers
For the Embroider Initiative to be successful, it needs backing from companies
that would benefit from a modern and thriving Ember ecosystem. But the benefits
of sponsoring the project go beyond the end goal.
While all sponsorship tiers (starting at 3k€ for individual supporters) include
a backer's logo in Mainmatter communication around the Initiative, companies
committing 18k€ to the project get access to the weekly 1-hour call with the
team and get opportunities to discuss their technical needs and challenges
related to Embroider.
Premium tiers of 36k€ and above include a 2-hour weekly private session with
Mainmatter engineers focused on improving the backer's Ember build. In practice,
these backers have seen a great return on their investment as the Mainmatter
Embroider team has been able to deliver dramatic speed improvements to their
build, removing a bottleneck in their development process. Some issues specific
to these backers also resulted in solutions that the larger Ember community can
now benefit from.
Let's see some success stories in details:
Ticketsolve
Like many companies, Ticketsolve has an internal addon that they use throughout
their other applications. Internal addons are a great place to start when you
are thinking about upgrading to Embroider for two reasons:

1. It's an isolated place to opt into Embroider's stricter requirements
2. There is a
   clear specification
   of what your v2 addon can do.

Chris' work on improving the ergonomics of the v2 addon blueprint in the early
weeks of the Embroider Initiative, proved very useful during the process of
converting Ticketsolve's internal addon.
After the internal addon was fully converted to v2 and deployed to all of
Ticketsolve's apps, our next step was to add support for
GJS files
in both the internal addon and their applications. The team also worked on the
ecosystem PRs required to enable GJS support in v2 addons in both a
@embroider/addon-dev
PR and a
PR to the
@embroider/addon-blueprint.
Before the Embroider Initiative, Ticketsolve had one of their three apps already
running Embroider. We worked to update the Embroider version of that app, then
started to convert the remaining two apps. There were some challenges along the
way, mostly related to addons that just didn't support Embroider. A great
example of this is the
ember-service-worker addon
which uses the postprocessTree() hook that doesn't work with Embroider. We
updated the addon locally using pnpm patch
and
submitted a PR with the same changes
to help the rest of the community.
This is only a fraction of the changes that came out of these pairing sessions,
but it gives a flavour of the kinds of things that we have been able to achieve.
While our Chris Manson has been instrumental in helping speed up this work,
Ticketsolve's engineer Andreas Minnich deserves a
shout-out for the colossal amount of work they have put into this effort between
pairing sessions.
Intercom
Intercom also had an internal addon they wanted to convert to v2 before working
on converting their main application. Mainmatter worked with them to convert
that and, incidentally, we also helped them to convert the tests app that was
documenting the v2 addon to use Embroider with the staticAddonTestSupportTrees
and staticAddonTrees flags turned on.
While this work was going on, Intercom Engineer
Aaron Chambers had setup a CI build step that
would keep track if the main Intercom app could build with Embroider and if any
tests were passing. Because of this CI job, Aaron identified a 10x slowdown in
build times between Embroider v3.1 and v3.2. Chris used a number of pairing
sessions to dive into the issue and produce a number of flamecharts for their
Embroider builds to get an insight into what was causing their specific
slowdown. Those flamecharts were pointing at a particular part of Embroider's
package cache not working exactly as we intended it to. Ed Faulkner managed to
pinpoint the problem (while we were discussing it in the "hallway track" of
EmberFest) and
opened a pull-request with the fix.
This completely fixed the performance regression for Intercom.
The Mainmatter Embroider team also noticed that Intercom's build was behaving
differently on CI compared to running the exact same build locally. We tracked
it down to the fact that Intercom's CI was configured to create a symbolic link
to an existing node_modules folder rather than running a new npm install for
every job. As it turns out, this was the same reason that the CI was failing for
Chris' pull request to make Embroider optimised the default
when running the command ember new --embroider. Chris identified the cause of
the issue inside Embroider and
fixed it without the
need for any assistance from Ed, which also shows the success in making
Embroider more maintainable by increasing the team size and distributing
technical knowledge (more about this in the next section).
The rest of the pairing sessions were spent working through issues that were
causing tests to fail. Some of these issues were systemic and when they were
fixed they would turn hundreds of tests green (Intercom has a lot of tests),
while other problems were highly specific and required multiple pairing sessions
to identify and fix the problem. An example of this would be Intercom's use of
the charting library Highcharts. When a Highcharts
user needs to include timezones in their Date axis, the library will check for
the presence of window.moment and use it for any timezone-related
calculations. This allows the user to have a chance to setup window.moment to
use moment-timezone and correctly configure it with the right set of timezone
data for their application. With the move to Embroider, window.moment is no
longer accidentally set to the correct instance of moment-timzone as a
side-effect of the build system, so Highcharts wasn't finding the right instance
during its initialisation. It turns out that Highcharts
provides a config option,
time.moment, to cover this exact case and, as soon as the team set that
correctly in the application's charts base class, the x-axis started behaving
again.
Again, these are only a fraction of the changes that came out of these pairing
sessions. Both Aaron Chambers and
Peter Meehan were able to achieve a lot of big things
over the course of the Embroider Initiative for Intercom and our team was more
than happy to help speed the process along.
Improving the bus factor
The goals of Embroider may seem simple from the outside, i.e. "just use Webpack
or Vite to build your Ember app", but when digging a bit deeper, it's easy to
see how complex of a project it really is. This complexity arises from the
project having challenging design constraints which pose a significant challenge
to anyone who would like to contribute to the core of Embroider. The main design
constraint that causes a lot of this complexity arises from the fact that the
team wants to provide an easy on-ramp for existing Ember apps to convert to
Embroider, and then slowly move those apps from full-compatibility mode to a
"fully static" build that can automatically benefit from tree-shaking and
code-splitting. This means that we need to provide systems that can
automatically convert the still-supported conventions of an Ember app to fully
standard compliant ESM code. This is a significant challenge since some of the
patterns that are still officially supported today date back to the 1.x
series of Ember which was released in 2015.
Because of the overall complexity of project and the rapid iteration of the
internal architecture, it hasn't been practical to create any documentation that
could communicate what is going on under the hood. This is illustrated by the
fact that Aaron Chambers and
Peter Meehan put together a
document trying to explain the full architecture of the project,
which was completely outdated only less than a month later when Ed Faulkner
merged
the first of many internal refactors.
To address this challenge, we have adopted an apprenticeship model where
Ed Faulkner pairs with Chris every week,
collaborating on solving complex problems deep within the heart of the Embroider
codebase. This has made a significant improvement to the
bus factor of the Embroider project
and has also had a deep impact on its velocity.
With our second full-time engineer on the Embroider Initiative,
Andrey Mikhaylov, we have extended the
apprenticeship model by having Chris and Andrey pair-program for half a day each
week. This has helped both Chris and Andrey ramp up their knowledge of the
codebase while improving the overall project's bus factor.
General stability and ecosystem improvements
Providing watch-mode tests for Embroider
Recently, Godfrey Chan has been discovering
some places in Embroider and our Webpack plugin that were
causing crashes when certain files were added or deleted.
He has already come up with a
fix for some of the cases,
but it showed a blind spot in the team's testing infrastructure that meant we
weren't testing "watch mode" in Embroider.
Preston Sego and Chris paired together to
add some basic watch mode tests
that would highlight the problem and prove the effectiveness of the fix, but
they were hit by some strange quirks that prevented the watch-mode tests from
ever exiting properly on the Windows CI job. Chris spent most of his
weekly streaming session on Twitch trying to
figure out the solution.
The team finally got the PR merged and is now ready to start adding more
expansive watch-mode tests.
ember-cli-update supporting v2 addons
ember-cli-update is a great asset to the Ember community, and it's a vital
tool during the push for Embroider and encouraging the community to use the new
v2 addon blueprint because a lot of the recent changes require updating both a
dependency version and related configuration.
One issue the team worked through was to fix CI, which had been broken for a
year. ember-cli-update interacts with the npm APIs to discover versions, but a
change on the npm side prevented it from functioning properly. The solution to
fixing CI was ultimately updating a dependency, boilerplate-update, to use
pacote to interact with the npm API.
Once the upstream dependency was updated, the team only needed to
add a few tweaks to ember-cli-update
to get CI to finally pass. This gave us time and confidence for the rest of the
project.
The team then worked to get a new release of
ember-cli-update so that it
could update any Ember addon generated with the
v2 addon blueprint.
ember-cli-update works by generating a diff between the version of the
blueprint that the app or addon was generated with, and the version to update it
to. Under the hood, this means that ember-cli-update will generate a pristine
new version of the app/addon at its current version, then generates a new
pristine copy of the target version. It generates a git diff between those
versions then applies that diff to the current app.
Generating these pristine "from" and "to" versions of the blueprint involved a
custom code-path in ember-cli-update that aimed at working around some bugs in
Node 8. As those workarounds were causing it to fail, Chris'
fix for custom blueprints
involved removing these workarounds.
This fix was released in ember-cli-update v2.0.0,
then followed by
another release
that fixed a small bug. These two releases are a big deal for the wider Ember
community, especially as its members are encouraged to migrate their existing v1
addons to v2 addons. They bring an important DX functionality that the Ember
community has come to expect.
Embroider optimised with the --embroider flag
Creating a new Ember app with the embroider flag
(ember new my-super-app --embroider) generates a "full compat" app were none
of the
Embroider optimisation flags
are passed to the build for the developer.
While more existing apps will work with a "full compat" mode, Ember has reached
a point where it makes sense for newly generated apps to start with a high
water mark so that developers don't accidentally or unconsciously add
functionality that won't work in a fully optimised Embroider application.
Developers have the option to turn off any of the optimisation flags, but it
would be a deliberate choice as they would need to add a specific dependency or
functionality to their app.
The Mainmatter Embroider team opened a
PR to switch the functionality
so that passing the --embroider flag uses Embroider optimised by default. This
involved working through issues with some of the slow test suites that rely on a
custom package caching mechanism. The PR got merged and the functionality will
be part of Ember v5.5.
scenario-tester ESM compatibility
Scenario tester is a
testing tool that the team makes use of a lot when testing Embroider and
ember-auto-import. It allows us to generate many scenarios with different
combinations of dependencies. We tried using it outside of Embroider, or more
specifically outside of a Typescript project, and saw it doesn't work in a CJS
environment.
Chris started the effort to move the Typescript build to output an ESM-compatible build
so it can be consumed directly in ESM without a build step. The only remaining
task is to test if it will work in Embroider before merging and releasing the
new version.
Documenting the scenario-tester library.
scenario-tester is to Embroider
what ember-try is to Ember CLI: it's a tool that lets us perform automated
tests with various combinations of dependencies, configs and circumstances. The
approach of scenario-tester is different: instead of reinstalling dependencies
for every test case, it has all dependencies (including all versions of
dependencies) set up once, saving a lot of time. It leverages
fixturify-project to
create and emit to filesystem Ember apps and addons with predefined dependencies
and configuration, in order to run tests on them.
Working on the reverse-exports
During the build, Embroider needs to expose Ember internals to Vite and Webpack
in a way they can understand and consume. Modern Ember apps can have multiple
exports entry
points in their package.json configs. This poses a peculiar challenge for
Embroider: it needs to know how to reorganize files in an Ember project in such
a way that they would resolve into paths defined as exports values.
Essentially, this requires resolving exports in reverse, and this it what the
reverse-exports
package is for.
Call to action
We are hoping we can extend the initiative's budget and timeline to keep up the
momentum in 2024 and finish the work we started. Please get in touch
to learn how to support the Embroider Initiative and directly benefit from this
investment!