Use standalone .js bundle in dist tarball rather than individual JS files #3030

Daniel15 · 2017-04-01T21:06:47Z

Summary

Instead of including all the raw JS files in the dist tarball, just use the single Yarn JS file that's built as part of the build, along with a few other files that are required. This significantly reduces the number of files in the tarball:

C:\src\yarn\dist (bundle-as-dist) ([email protected])
λ find .
.
./bin
./bin/node-gyp-bin
./bin/node-gyp-bin/node-gyp
./bin/node-gyp-bin/node-gyp.cmd
./bin/yarn
./bin/yarn.cmd
./bin/yarn.js
./bin/yarnpkg
./bin/yarnpkg.cmd
./lib
./lib/v8-compile-cache.js
./lib/yarn-cli.js
./LICENSE
./package.json

There are three .js files in the archive:

lib/v8-compile-cache.js: Speeds up instantiation time by using the V8 code cache (https://www.npmjs.com/package/v8-compile-cache). This needs to be separate as it has to load before the bulk of the application code is loaded, so it can not be bundled
lib/yarn-cli.js: Contains all the bundled Yarn code
bin/yarn.js: Entry point to the app, just like today. Loads v8-compile-cache then loads yarn-cli

This change means that only the JavaScript files that are actually used are included, resulting in a nice file size reduction for the installation packages:

Differences are due to differing compression algorithms: Debian packages use xz or LZMA, RedHat uses gzip, Windows installer uses Cabinet

They're also slightly faster to extract:

Testing was performed on my desktop computer (Intel Core i5 6500, Samsung 850 Evo 1TB SSD, Windows 10), with testing for Linux stuff (like installing the Debian package) tested in a Docker container.

Raw data: https://docs.google.com/spreadsheets/d/1d8jdf3DU_GUFdotlPl08PkYa8SkzStK2tgnQ54ivsm0/edit?usp=sharing

Performance is very slightly faster when using v8-compile-cache along with the bundled file, but it's not extremely significant (yarn --version went from 0.19s to 0.14s on my BuyVM server). The difference might be bigger on servers with slower disks (HDD) or with more overloaded servers.

I also deleted the build-dist.ps1 file because we should be able to assume that Bash is available on Windows, particularly if Git is installed (as it comes with Git Bash). I need to verify that this works on AppVeyor.

Test plan
I tested the Debian package and Windows installer on my system. Both work as expected. I also manually inspected the tarball to ensure files were in the right place and bin/yarn still works as expected.

Wait for CI to pass before merging!

Closes #2966

cc @yarnpkg/core

Daniel15 · 2017-04-01T21:18:22Z

The Windows build is getting "stdout is not a tty" but it works fine on my computer... Will need to take a closer look.

bestander · 2017-04-01T22:15:57Z

Oh My!

…

On Sat, 1 Apr 2017 at 22:07, Daniel Lo Nigro ***@***.***> wrote: *Summary* Instead of including all the raw JS files in the dist tarball, just use the single Yarn JS file that's built as part of the build, along with a few other files that are required. This significantly reduces the number of files in the tarball: C:\src\yarn\dist (bundle-as-dist) ***@***.***) λ find . . ./bin ./bin/node-gyp-bin ./bin/node-gyp-bin/node-gyp ./bin/node-gyp-bin/node-gyp.cmd ./bin/yarn ./bin/yarn.cmd ./bin/yarn.js ./bin/yarnpkg ./bin/yarnpkg.cmd ./lib ./lib/v8-compile-cache.js ./lib/yarn-cli.js ./LICENSE ./package.json There are three .js files in the archive: - lib/v8-compile-cache.js: Speeds up instantiation time by using the V8 code cache (https://www.npmjs.com/package/v8-compile-cache). This needs to be separate as it has to load *before* the bulk of the application code is loaded, so it can *not* be bundled - lib/yarn-cli.js: Contains all the bundled Yarn code - bin/yarn.js: Entry point to the app, just like today. Loads v8-compile-cache then loads yarn-cli This change means that *only* the JavaScript files that are actually used are included, resulting in a nice file size reduction for the installation packages: <https://camo.githubusercontent.com/d3c3254ad95f9e4204f97292063ad27b9779b9da/687474703a2f2f73732e64616e2e63782f323031372f30342f5961726e5f62756e646c655f646973745f6d6574726963735f2d5f476f6f676c655f5368656574735f2d5f476f6f676c655f5f30312d31332e35312e34392e706e67> Differences are due to differing compression algorithms: Debian packages use xz or LZMA, RedHat uses gzip, Windows installer uses Cabinet They're also slightly faster to extract: [image: image 3] <https://cloud.githubusercontent.com/assets/91933/24582332/483b41f4-16e2-11e7-9509-8024b1e78a39.png> Testing was performed on my desktop computer (Intel Core i5 6500, Samsung 850 Evo 1TB SSD, Windows 10), with testing for Linux stuff (like installing the Debian package) tested in a Docker container. Raw data: https://docs.google.com/spreadsheets/d/1d8jdf3DU_GUFdotlPl08PkYa8SkzStK2tgnQ54ivsm0/edit?usp=sharing Performance is very slightly faster when using v8-compile-cache along with the bundled file, but it's not extremely significant (yarn --version went from 0.19s to 0.14s on my BuyVM server). The difference might be bigger on servers with slower disks (HDD) or with more overloaded servers. I also deleted the build-dist.ps1 file because we *should* be able to assume that Bash is available on Windows, particularly if Git is installed (as it comes with Git Bash). I need to verify that this works on AppVeyor. *Test plan* I tested the Debian package and Windows installer on my system. Both work as expected. I also manually inspected the tarball to ensure files were in the right place and bin/yarn still works as expected. Wait for CI to pass before merging! cc @yarnpkg/core <https://github.com/orgs/yarnpkg/teams/core> ------------------------------ You can view, comment on, or merge this pull request online at: #3030 Commit Summary - Use standalone .js bundle in dist tarball rather than individual JS files File Changes - *M* appveyor.yml <https://github.com/yarnpkg/yarn/pull/3030/files#diff-0> (5) - *A* bin/yarn-bundle-entry.js <https://github.com/yarnpkg/yarn/pull/3030/files#diff-1> (7) - *M* circle.yml <https://github.com/yarnpkg/yarn/pull/3030/files#diff-2> (5) - *M* package.json <https://github.com/yarnpkg/yarn/pull/3030/files#diff-3> (4) - *M* scripts/build-deb.sh <https://github.com/yarnpkg/yarn/pull/3030/files#diff-4> (16) - *D* scripts/build-dist.ps1 <https://github.com/yarnpkg/yarn/pull/3030/files#diff-5> (18) - *M* scripts/build-dist.sh <https://github.com/yarnpkg/yarn/pull/3030/files#diff-6> (43) - *D* scripts/clean-node-modules.ps1 <https://github.com/yarnpkg/yarn/pull/3030/files#diff-7> (11) - *D* scripts/clean-node-modules.sh <https://github.com/yarnpkg/yarn/pull/3030/files#diff-8> (10) Patch Links: - https://github.com/yarnpkg/yarn/pull/3030.patch - https://github.com/yarnpkg/yarn/pull/3030.diff — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#3030>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBdWL8nbet4IhWKu5_AfOYLPs0Mc_dNks5rrrx2gaJpZM4MwnU5> .

cpojer · 2017-04-01T23:17:12Z

Wow, great work, man!

bestander · 2017-04-02T23:14:09Z

Love it so much!
Let me know when ready to merge.
What do you think of getting rid of roadrunner as well?
The extra cache layer for people who run from master branch does not make much impact.

Daniel15 · 2017-04-04T21:04:15Z

It looks like there's a build error in Node.js 4.x as build-webpack.js is using syntax it doesn't support: https://ci.appveyor.com/project/kittens/yarn/build/1942/job/33t035v195s3k19f

I'll update that, and then this should be ready to merge.

cdaringe · 2017-04-07T22:07:43Z

does this affect:

support requests? if we use a bundle, the trace may be less helpful in the generated log file
debug-ability?

w/ how many issues are open ATM, it would make sense to me that support take precedence over these perf improvements (which do look great, btw).

just sharing my opinion!

Daniel15 · 2017-04-07T22:09:59Z

debug-ability?

People that are debugging Yarn itself would probably be using Yarn directly from this Git repo, which isn't affected by these changes

support requests? if we use a bundle, the trace may be less helpful in the generated log file

This is one thing I was worried about, however all the function names remain intact (no minification is performed) so stack traces shouldn't be too bad.

We could ship two tarballs, Debian packages and Windows installers if needed - a "debug" one that's like the tarball today, and a "release" one that's the smaller combined source file.

bestander · 2017-04-07T22:39:28Z

From my experience of debugging Yarn a lot, a single file build almost never was a problem

…iles

…indows Git Bash

…parately.

Daniel15 · 2017-04-08T21:38:29Z

I'm going to merge this so we can get it into the next RC. We can iterate as needed. The AppVeyor build is passing, and the CircleCI failure is due to unrelated test timeouts (the actual build itself passed).

bestander · 2017-04-08T21:42:06Z

scripts/build-dist.sh

+cp artifacts/yarn-legacy-*.js dist/lib/yarn-cli.js
+cp bin/yarn-bundle-entry.js dist/bin/yarn.js
+cp bin/{yarn,yarnpkg,*.cmd} dist/bin/
+cp -r bin/node-gyp-bin dist/bin/


Do we want to copy node-gyp from node_modules, too?

Ahh, good catch! Do we need it? I tried yarn install bcrypt and it seemed to work, but perhaps I have a globally-installed node-gyp or something like that. I can test on a fresh VM and see if it works.

Actually I wonder if Yarn should install node-gyp when needed (ie. run yarn global add node-gyp) rather than bundling it? I think bundling node-gyp would also require us to bundle all of its dependencies, so we'll end up with a giant node_modules directory bundled with Yarn again :(

I agree.
That actually will solve the issue of single js file distributions.
Maybe Yarn could check if node-gyp binary is available and if not then install it.

The down side is that in some cases we Yarn might need sudo permissions?
What if Yarn would install node-gyp locally next to yarn.js?

Does this change anything about #2266?

@teohhanhui - We'll most likely update Yarn to automatically install node-gyp if needed (see #3114)

This was referenced Apr 7, 2017

Roadrunner cache occupies 1GB of RAM after de-serialization #3037

Closed

Checks that the webpack builds are working properly #3064

Merged

Daniel15 added 6 commits April 8, 2017 13:41

Use standalone .js bundle in dist tarball rather than individual JS f…

3b0fdde

…iles

Work around #2591 by explicitly executing yarn.cmd when building on W…

5916f40

…indows Git Bash

Only run shasum if it's installed

ac47aa9

Remove shasum from dist build script altogether - It can be hashed se…

0909197

…parately.

Update build-webpack to work on Node.js 4.x

185c5d0

Ensure bundled JS works as expected (similar to #3064)

e58ddf2

Daniel15 force-pushed the bundle-as-dist branch from f21d0ca to e58ddf2 Compare April 8, 2017 20:46

Daniel15 added 2 commits April 8, 2017 13:55

Include original build-dist script as build-dist-debug.sh

8bda8b9

Adjust how version number is retrieved in CircleCI build

421796c

Daniel15 force-pushed the bundle-as-dist branch from 10200a2 to 421796c Compare April 8, 2017 21:20

Daniel15 merged commit 3af60cf into master Apr 8, 2017

Daniel15 deleted the bundle-as-dist branch April 8, 2017 21:40

bestander reviewed Apr 8, 2017

View reviewed changes

Daniel15 mentioned this pull request Apr 8, 2017

Remove Roadrunner #3079

Merged

Daniel15 mentioned this pull request Apr 23, 2017

Remove junk from dist #908

Closed

Daniel15 mentioned this pull request May 4, 2017

Error "/usr/bin/yarn: [[: not found" with 0.24.0 #3321

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use standalone .js bundle in dist tarball rather than individual JS files #3030

Use standalone .js bundle in dist tarball rather than individual JS files #3030

Daniel15 commented Apr 1, 2017 •

edited

Loading

Daniel15 commented Apr 1, 2017

bestander commented Apr 1, 2017 via email

cpojer commented Apr 1, 2017 via email •

edited by arcanis

Loading

bestander commented Apr 2, 2017

Daniel15 commented Apr 4, 2017

cdaringe commented Apr 7, 2017

Daniel15 commented Apr 7, 2017

bestander commented Apr 7, 2017

Daniel15 commented Apr 8, 2017

bestander Apr 8, 2017

Daniel15 Apr 8, 2017

Daniel15 Apr 8, 2017

bestander Apr 8, 2017

bestander Apr 8, 2017

teohhanhui Apr 13, 2017

Daniel15 Apr 13, 2017

Use standalone .js bundle in dist tarball rather than individual JS files #3030

Use standalone .js bundle in dist tarball rather than individual JS files #3030

Conversation

Daniel15 commented Apr 1, 2017 • edited Loading

Daniel15 commented Apr 1, 2017

bestander commented Apr 1, 2017 via email

cpojer commented Apr 1, 2017 via email • edited by arcanis Loading

bestander commented Apr 2, 2017

Daniel15 commented Apr 4, 2017

cdaringe commented Apr 7, 2017

Daniel15 commented Apr 7, 2017

bestander commented Apr 7, 2017

Daniel15 commented Apr 8, 2017

bestander Apr 8, 2017

Choose a reason for hiding this comment

Daniel15 Apr 8, 2017

Choose a reason for hiding this comment

Daniel15 Apr 8, 2017

Choose a reason for hiding this comment

bestander Apr 8, 2017

Choose a reason for hiding this comment

bestander Apr 8, 2017

Choose a reason for hiding this comment

teohhanhui Apr 13, 2017

Choose a reason for hiding this comment

Daniel15 Apr 13, 2017

Choose a reason for hiding this comment

Daniel15 commented Apr 1, 2017 •

edited

Loading

cpojer commented Apr 1, 2017 via email •

edited by arcanis

Loading