The future of create-esm-loader

[sebamarynissen]

The future of create-esm-loader

TL;DR You should no longer rely on this module for being able to combine loaders, and just write native customization hooks instead.

A bit of history

When Node unflagged ES Modules in v14, they provided an initial set of loader hooks to make it possible to import non-js files, just like you could do with require.extensions. However, it was explicitly stated that this API was experimental and would still change in the future. There was also no way to combine multiple loaders: you could only export one set of hooks, so combining loaders was something you had to do yourself.

The goal of create-esm-loader was to make it possible to combine loaders and to limit the impact of breaking changes in the loaders api. The idea was that if the loaders api changed, only this module had to be updated, without the need for rewriting your loaders.

The module has done this fairly well: when the getFormat(), getSource() en transformSource() hooks were removed in favor of a single load() hook in Node 16.12, you didn't have to rewrite your loaders. Just updating this module and then changing

import create from 'create-esm-loader';
export const { resolve, getFormat, getSource, transformSource } = await create(config);

to

import create from 'create-esm-loader';
export const { resolve, load } = await create(config);

was sufficient. You could even support both Node version before 16.12 and above by exporting getFormat, getSource and transformSource along with load.

In the meantime, the loaders api has more or less stabilized and Node has added the possibity to combine loaders with module.register():

import { register } from 'node:module';

register('./foo.js', import.meta.url);
register('./bar.js', import.meta.url);

This means that the two main reasons of why create-esm-loader was conceived are now no longer there: no more breaking changes in the loaders api are expected, and it is possible to natively combine multiple loaders. Therefore, if you've written loaders on top of this module, you should change them to use the native resolve(), load() and initialize() customization hooks instead.

Migrating to native customization hooks

If you've written a loader on top of create-esm-loader, there is some work to migrate to native customization hooks, but it's definitely doable and even makes the loaders more readable. Most loaders do some kind of source transform, which means all you have to do is export a load() hook. For example, a simpe TypeScript loader could look like this

// # typescript-loader.js
import esbuild from 'esbuild';

export async function load(url, ctx, nextLoad) {

  // If the url does not end with `.ts`, skip to the next loader in the chain.
  if (!url.endsWith('.ts')) return nextLoad(url);

  // Use the next loaders to get the source, which defaults to Node loading
  // the file from disk, but there can also be a loader that fetches from the
  // network for example further down the chain. It's important that you
  // specify the "format" here, because Node doesn't know that `.ts` files
  // should be treated as esm!
  let { source } = await nextLoad(url, { format: 'module' });

  // Strip the types with esbuild and return.
  let { code } = await esbuild.transform(source, { loader: 'ts' });
  return {
    source: code,
    format: 'module',
  };

}

// Use it as
import { register } from 'node:module';
register('./typescript-loader.js', import.meta.url);

There is no need anymore to implement a custom resolve() hook as we just use Node's default resolution algorithm for .ts files.

If you've written a loader that modifies the resolution algorithm, for example to allow you to do something like

import fn from '@/helpers/fn.js';

then all you have to do is export a resolve() hook

// # cwd-loader.js
import path from 'node:path';
import { pathToFileURL } from 'node:url';

export async function resolve(specifier, ctx, nextResolve) {

  // Only handle specifiers that start with `@/`, otherwise let the default 
  // resolution algorithm handle it.
  if (!specifier.startsWith('@/')) return nextResolve(specifier);

  // Treat @ as the current working directory.
  let rest = specifier.slice(2);
  let file = path.join(process.cwd(), rest);
  let url = pathToFileURL(file).href;

  // shortCircuit: true is needed to signal to node that we intentionally 
  // did not use the default resolution algorithm.
  return {
    url,
    shortCircuit: true,
  };

}

Combining the TypeScript loader and the @-loader can then be done with

import { register } from 'node:module';
register('./typescript-loader.js', import.meta.url);
register('./cwd-loader.js', import.meta.url);

after which

import '@/src/file.ts';

becomes possible.

Often you want to be able to configure your loaders with options as well, for example to pass in an array of file extensions. This can be done with the initialize() hook:

// # options-loader.js
let extension;
export function initialize(data) {
  extension = data.extension;
}

export async function load(url, ctx, nextLoad) {
  if (!url.endsWith(extension)) return nextLoad(url);

  // Custom source transform here.
  return {
    source: '...',
  };

}

which can be used as

import { register } from 'node:module';
register('./options-loader.js', import.meta.url, {
  data: {
    extension: '.ts',
  },
});

The future of this module

While you definitely don't need this module anymore to write composable loaders, I still think there's a place for it with a slightly different scope. I'm planning to rewrite the module - as a semver major obviously - so that it can still remove a bit of the boilerplate needed for writing common loaders.

For example, if you just want a source transform, you could write

import create from 'create-esm-loader';

export const { load } = create({
  test: /\.ext$/,
  async transform(source) {
    return await transform(source);
  },
});

Alternatively, it could also be used to make accessing options easier:

import create from 'create-esm-loader';

// The initialize hook is created automatically and sets `this.options`.
export const { load, initialize } = create({

  // `resolve()` has the exact same signature as the resolve hook, but with
  // access to this.options
  async resolve(specifier, ctx, nextResolve) {
    const alias = this.options;
    if (specifier in alias) {
      return nextResolve(alias[specifier]);
    }
    return nextResolve(specifier);
  },

  // Same for `load`: same signature as the load hook, but with access
  // to options.
  async load(url, ctx, nextLoad) {
    const { extension } = this.options;
    if (!new URL(url).pathname.endsWith(extension)) return nextLoad(url);
    return {
      source: '...',
      format: 'module',
    };
  },

});

It might even be possible to export register() as well

// # loader.js
import create from 'create-esm-loader';

export const { initialize, resolve, load, register } = create({
  resolve(specifier, ctx, nextResolve) {},
  load(url, ctx, nextLoad) {},
});

so that you can register the loader as

// # register-loader.js
import { register } from './loader.js';
register({
  some: 'options',
});

instead of doing

// # register-loader.js
import { register } from 'node:module';
register('./loader.js', import.meta.url, {
  data: {
    some: 'options',
  },
});

[sebamarynissen]

@brev I thought you might be interested in this one. Feel free to tag other people that you know have written loaders on top of this module.