serverless + esbuild + npm not including required platform dependencies

[demian85]

Question about an existing feature

What are you trying to achieve?

Trying to deploy a Lambda function using Serverless v4 + esbuild + npm v10

When you searched for similar issues, what did you find that might be related?

I'm having issues including the optional/required binary platform-dependent dependencies to make the lib work. None of the documented instructions on the cross-platform section helped overcome this issue.
The error I get while executing the lambda is Could not load the "sharp" module using the linux-x64 runtime.
After following all the instructions in the docs, the only thing that worked for me is adding the following lines in the serverless.yml file, but I can't make sense of it.

package:
  individually: true
  patterns:
    - 'node_modules/@img/sharp-libvips-linux-x64'

• Why isn't esbuild including the required platform specific dependencies itself?
• Why do I need to hard-code the inclusion of this platform-specific module in all the functions I deploy? I'm deploying like 80 lambdas and now all of them now include @img/sharp-libvips-linux-x64 as a dependency?
• Why do I need to exclude sharp from bundling? How does the lambda function find sharp if it is not in the bundle?
• I can see in the deployment package that sharp is in the node_modules of every other lambda function that does not even use sharp at all. How's that?

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this question

Just try to resize an image using sharp. Runtime error is thrown:

"stack": [
        "Error: Could not load the \"sharp\" module using the linux-x64 runtime",
        "Possible solutions:",
        "- Ensure optional dependencies can be installed:",
        "    npm install --include=optional sharp",
        "- Ensure your package manager supports multi-platform installation:",
        "    See https://sharp.pixelplumbing.com/install#cross-platform",
        "- Add platform-specific dependencies:",
        "    npm install --os=linux --cpu=x64 sharp",
        "- Consult the installation documentation:",
        "    See https://sharp.pixelplumbing.com/install",
        "    at <anonymous> (/node_modules/sharp/lib/sharp.js:113:9)",
        "    at /var/task/src/lambda/files/upload.js:1:258",
        "    at <anonymous> (/node_modules/sharp/lib/constructor.js:10:1)",
        "    at /var/task/src/lambda/files/upload.js:1:258",
        "    at <anonymous> (/node_modules/sharp/lib/index.js:6:15)",
        "    at /var/task/src/lambda/files/upload.js:1:258",
        "    at Object.<anonymous> (/src/lambda/files/upload.ts:10:19)",
        "    at Module._compile (node:internal/modules/cjs/loader:1358:14)",
        "    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)",
        "    at Module.load (node:internal/modules/cjs/loader:1208:32)"
    ]

Please provide sample image(s) that help explain this question

?

Thanks!

[lovell]

Did you see https://sharp.pixelplumbing.com/install#esbuild and the serverless esbuild example? Watch out for the npm lockfile bug npm/cli#4828 too.

[demian85]

The serverless esbuild example is meant for v3. v4 uses esbuild config file instead of a plugin, so it confuses me.
I already installed the optional platform specific dependencies, I can see them in the package-lock.json file and somehow it worked when deploying from Github actions.
My questions still remain.

[lovell]

https://sharp.pixelplumbing.com/install#esbuild

Ensure sharp is excluded from bundling via the external configuration.

https://www.serverless.com/framework/docs/providers/aws/guide/building#configuration

V.4 introduces a new build configuration block, which you can use to customize esbuild settings.

Putting these two together would suggest something like the following is required:

build:
  esbuild:
    external:
      - sharp

[demian85]

I already did that and didn't make a difference. The only thing that worked is using the patterns config in serverless.yml.
The point of this issue is just to ask all those questions in the first post because I'm trying to make sense of this.
If sharp is excluded, why does it appear in the node_modules package?

[lovell]

Why isn't esbuild including the required platform specific dependencies itself?

Why do I need to exclude sharp from bundling?

esbuild doesn't support bundling native modules. My understanding is that this is intentional, which makes sense as it's relatively easy to mark packages as external to bundling.

Why do I need to hard-code the inclusion of this platform-specific module in all the functions I deploy?

This sounds like a limitation of the serverless tooling. If you mark packages as external at bundle time then it would make sense for these to be automagically included at package time, but that doesn't (yet) appear to be a feature.

How does the lambda function find sharp if it is not in the bundle?

Native modules should be packaged but not bundled, and your deployment zip file should contain a node_modules directory with native modules.

I can see in the deployment package that sharp is in the node_modules of every other lambda function that does not even use sharp at all. How's that?

My best guess would be because the serverless configuration you've provided suggests you're telling it to always include it via package.patterns for all functions.

https://www.serverless.com/framework/docs/providers/aws/guide/packaging#individual-function-packages suggests the following might be more appropriate:

package:
  individually: true
functions:
  my-function-that-uses-sharp:
    package:
      patterns:
        - 'node_modules/sharp'
        - 'node_modules/@img'

[demian85]

That makes sense now. Didn't know I could configure patterns per function. Thanks!

…

On Fri, Sep 6, 2024, 16:11 Lovell Fuller ***@***.***> wrote: Why isn't esbuild including the required platform specific dependencies itself? Why do I need to exclude sharp from bundling? esbuild doesn't support bundling native modules. My understanding is that this is intentional, which makes sense as it's relatively easy to mark packages as external to *bundling*. Why do I need to hard-code the inclusion of this platform-specific module in all the functions I deploy? This sounds like a limitation of the serverless tooling. If you mark packages as external at *bundle time* then it would make sense for these to be automagically included at *package time*, but that doesn't (yet) appear to be a feature. How does the lambda function find sharp if it is not in the bundle? Native modules should be *packaged* but not *bundled*, and your deployment zip file should contain a node_modules directory with native modules. I can see in the deployment package that sharp is in the node_modules of every other lambda function that does not even use sharp at all. How's that? My best guess would be because the serverless configuration you've provided suggests you're telling it to always include it via package.patterns for all functions. https://www.serverless.com/framework/docs/providers/aws/guide/packaging#individual-function-packages suggests the following might be more appropriate: package: individually: truefunctions: my-function-that-uses-sharp: package: patterns: - 'node_modules/sharp' - ***@***.***' — Reply to this email directly, view it on GitHub <#4213 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADS4ZWKIDIAF7SKTLJDTTTZVH46HAVCNFSM6AAAAABNYOSMQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZUGY3DINJRGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[lovell]

I hope this information helped. Please feel free to re-open with more details if further assistance is required.