Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run compress_assets without management command (faster, faster) #2000

Open
KevinMind opened this issue Mar 22, 2024 · 9 comments · May be fixed by mozilla/addons-server#23043
Open

Run compress_assets without management command (faster, faster) #2000

KevinMind opened this issue Mar 22, 2024 · 9 comments · May be fixed by mozilla/addons-server#23043
Assignees
Labels
repository:addons-server Issue relating to addons-server

Comments

@KevinMind
Copy link
Contributor

KevinMind commented Mar 22, 2024

Currently the command update_assets executes a lot of logic to copy, compress, and bundle static assets from our git repo and from node_modules. This logic is executed via make commands and some sub commands execute via django management commands.

This means in order to process our static assets we need to install all pip dependencies and make django management commands available.

This makes it difficult or impossible to execute the logic as a part of an isolated docker stage and instead run the logic after the rest of the docker build (current state). That adds something like 30 seconds to our docker build and is nearly impossible to cache because any change to any file will invalidate the cache.

Modifying this chain of logic to execute via shell scripts will significantly improve performance of our build.

┆Issue is synchronized with this Jira Task

@KevinMind
Copy link
Contributor Author

If we used something like https://django-compressor.readthedocs.io/en/stable/ then we could simply remove the compress assets management command and totally internalize the compression process.

There are a few compatibility issues though, so would need to verify this or some other equivalent package would even work.

@KevinMind
Copy link
Contributor Author

We could also consider bundling the assets using a javascript/web bundler like webpack/rollup/vite.

Instead of managing the bundles via MINIFIED_BUNDLES in django, we could simply reference build artifacts and let the npm tool do the building. This being isolated from django could be run in a separate stage and only depend on the static folder.

@diox
Copy link
Member

diox commented Mar 25, 2024

I am django-compressor maintainer, and I would recommend against using it. It's not meant for big/complex websites, and is not really in a place where it can compete with modern bundlers (I suspect all django packages for assets management suffer from the comparison with modern npm bundlers as well).

@KevinMind
Copy link
Contributor Author

TIL 👋

@eviljeff
Copy link
Member

We could also consider bundling the assets using a javascript/web bundler like webpack/rollup/vite.

Instead of managing the bundles via MINIFIED_BUNDLES in django, we could simply reference build artifacts and let the npm tool do the building. This being isolated from django could be run in a separate stage and only depend on the static folder.

https://github.com/MrBin99/django-vite fwiw (not checked out beyond the readme)

@KevinMind
Copy link
Contributor Author

One method I had played around with but not fully verified is something like

  1. replace MINIFY_BUNDLES dictionary with a manifest file, either yaml or json (this so we don't need django/python to read the assets
  2. define a rollup/webpack/vite build based on those configured bundles (like we have now with compress_assets just replacing the tool)
  3. make sure the path of the output bundles can be inferred by the input path, via a manifest or some heuristic
  4. update our jingo_minify_helpers to read the output file path via the output manifest.

This seems a lot like withat django-vite is doing, maybe? I'm fairly confident something like this means we can delete the compress_assets file and run the bundlificiationismination as a standalone stage and cache it to the depths of hell, unless you change an asset.

@KevinMind
Copy link
Contributor Author

@KevinMind KevinMind transferred this issue from mozilla/addons-server May 3, 2024
@KevinMind KevinMind added repository:addons-server Issue relating to addons-server migration:2024 labels May 3, 2024
@diox diox removed the needs:info label May 14, 2024
@eviljeff eviljeff removed their assignment Sep 19, 2024
@diox
Copy link
Member

diox commented Sep 25, 2024

Needs a spike to investigate this.

@KevinMind
Copy link
Contributor Author

Notes so far fyi @chrstinalin

  1. look at this ticket for mapping django template tags to jinja2 compliant tags
  2. look at this example repo for how to setup
  • we should define a docker-compose service for running vite that will compile all the assets during development.
  • this service should be available over docker network to nginx which can expose to open internet
  • we use the django_vite vite_asset to compute a script tag which resolves to the url -> nginx -> vite server asset directory.

Now loading assets in dev is almost the same as in prod. This "should" work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
repository:addons-server Issue relating to addons-server
Projects
None yet
3 participants