[useStorage] getItem and getItemRaw -> Inconsistent behaviour between dev and preview/production

[Zebnastien]

Environment

Operating System: Windows_NT
Node Version: v22.6.0
Nuxt Version: 3.12.4
Nitro Version: 2.9.7
Package Manager: [email protected]
Builder: vite
User Config: devtools, compatibilityVersion : 4
Runtime Modules: -
Build Modules: -

Context: Following "Assets" section in Nitro documentation to handle SQL files.

Reproduction

1 - Create server/assets/demo.sql with:

SELECT
    *
FROM
    projects;

2 - Create server/routes/test.get.ts with:

export default defineEventHandler(async (event) => {
    const sql = await useStorage("assets:server").getItem("demo.sql");
    const sqlRaw = await useStorage("assets:server").getItemRaw("demo.sql");
    const sqlRawToString = sqlRaw.toString();

    return {
        sql,
        sqlRaw,
        sqlRawToString,
    };
});

3 - Build the app and fetch this api both in dev and preview mode -> Results are inconsistents.

Describe the bug

Here is the result in dev mode:

{
  "sql": "SELECT\r\n *\r\nFROM\r\n projects;\r\n",
  "sqlRaw": {
    "type": "Buffer",
    "data": [
      83,
      69,
      76,
      ...
      59,
      13,
      10
    ]
  },
  "sqlRawToString": "SELECT\r\n *\r\nFROM\r\n projects;\r\n"
}

Here is the result in preview/production:

{
  "sql": {
    "0 ": 83,
    "1": 69,
    "2": 76,
    ...
    "33": 59,
    "34": 13,
    "35": 10
  },
  "sqlRaw": {
    "0": 83,
    "1": 69,
    "2": 76,
    ...
    "33": 59,
    "34": 13,
    "35": 10
  },
  "sqlRawToString": "83,69,76,...,59,13,10"
}

=> Results should be the same between dev and production mode.
In my particular case, those differences lead to database fetching error.

Additional context

Here is the content of .output/server/chunks/raw/demo.mjs:

function base64ToUint8Array(str) {
  const data = atob(str);
  const size = data.length;
  const bytes = new Uint8Array(size);
  for (let i = 0; i < size; i++) {
    bytes[i] = data.charCodeAt(i);
  }
  return bytes;
}

// ROLLUP_NO_REPLACE 
 const demo = base64ToUint8Array("U0VMRUNUDQogICAgKg0KRlJPTQ0KICAgIHByb2plY3RzOw0K");

export { demo as default };
//# sourceMappingURL=demo.mjs.map

Here is the content of .output/server/chunks/raw/demo.mjs if I rename demo.sql to demo.txt:

// ROLLUP_NO_REPLACE 
 const demo = "SELECT\r\n    *\r\nFROM\r\n    projects;\r\n";

export { demo as default };
//# sourceMappingURL=demo.mjs.map

The problem appears using whatever unsupported file extension leading to base64 encoding on build output.
No problem with txt, html or json files. Files with no extension can be retreived with getItem( ) in dev mode but are purely ignored from build (not present).

Also, might be related to #1926 and #2481

Logs

No response

[Zebnastien]

Whlite investigating, I found this related set of file extensions here:

nitro/src/rollup/plugins/raw.ts

Line 10 in af42c19

const extensions = new Set([

Appart from the issue described above, it might be useful to provide a way to edit this list through some config options to allow devs to choose how to handle custom file assets. If there already is a way, I could not figure it out.

[mbegerau]

I also encountered this difference using Nuxt today. In production you can get the string with a TextDecoder:

const sql = await useStorage("assets:server").getItem("demo.sql");
const sqlString = new TextDecoder().decode(sql);

But yes, the behavior should be equal in all environments. Because in dev this will throw an error The "list" argument must be an instance of SharedArrayBuffer, ArrayBuffer or ArrayBufferView. because sql is already a string.

[mbegerau]

I spent some time on this today. Actually the problem (or solution) is not the extension array which is used in the resolveId method. The culprit is the isBinary function that considers the file a binary file, even if you would add '.sql' to the extension array, because the mime library assigns "application/sql" as mime type for .sql files and the isBinary function interprets that as a binary file type.

So I found 3 workarounds to fix this for sql files:

1. Rename the file(s) from .sql to .txt and inform your IDE that these files should be considered as .sql files nonetheless. Most IDEs can override the language mode per file. (in vscode bottom right change from "Plain Text" to "MS SQL" or whatever).

2. Check the storage item type or check for the environment and decode if necessary. E. g.:

const sql = await useStorage<string | Uint8Array | null>("assets:server").getItem("demo.sql");
const sqlStringType = (typeof sql === 'string') ? sql : new TextDecoder().decode(sql); // I would prefer this
const sqlStringEnv = (process.env.NODE_ENV !== "production") ? sql : new TextDecoder().decode(sql); // Not type safe

3. Use a nitro hook to override the raw plugin configuration, and make the transform method think .sql files are .txt files (at your own risk!). Then you don't need any adjustments in the code, you can just assume the result from useStorage is of type string or null.
   e. g. in nuxt.config.ts

export default defineNuxtConfig({
    hooks: {
        'nitro:init': (nitro) => {
            nitro.hooks.hook('rollup:before', (_nitro, config: any) => {
                const raw = config.plugins.find(
                    (plugin: any) => plugin.name === 'raw'
                );
    
                const transform = raw.transform;
                raw.transform = (code: any, id: string) =>
                    transform(code, id.replace(/\.sql$/, '.txt'));
            });
        },
    },
});