Skip to content

bug: multer.memoryStorage() buffers entire upload in heap — concurrent large uploads cause OOM crash #12

Description

@brightpixel-dev

Severity

Critical — process crash under load / denial of service

Description

POST /api/ipfs/upload uses multer.memoryStorage(), which holds the entire uploaded file in a single Buffer on the Node.js heap until the upload to Pinata completes. With IPFS_MAX_FILE_MB defaulting to 100 MB, a small number of concurrent uploads will exhaust the process heap and trigger an out-of-memory (OOM) kill.

Code Evidence

```typescript
// src/routes/ipfs.ts:9
const upload = multer({ storage: multer.memoryStorage(), limits: { fileSize: MAX_FILE_BYTES } });
```

```typescript
// src/services/ipfs.ts:16-20
const res = await axios.post(PINATA_ENDPOINT, form, {
headers: { Authorization: Bearer ${PINATA_JWT}, ...form.getHeaders() },
maxContentLength: Infinity, // ← no upload size cap on the Axios side either
});
```

The upload flow is:

  1. Multer buffers 100 MB into req.file.buffer (heap allocation).
  2. uploadToIPFS appends that buffer to a FormData object (second copy in heap).
  3. Axios serialises the form before sending (potential third copy).

Peak heap cost per upload: ~300 MB. The default Node.js heap limit is ~1.5 GB.

```
Concurrent uploads | Peak heap usage
5 | ~1.5 GB → OOM threshold
10 | ~3.0 GB → guaranteed OOM kill
```

Impact

  • Five concurrent 100 MB uploads from different producers crash the process.
  • No graceful degradation — the process exits and all in-flight requests fail.
  • Pinata uploads that started before the crash leave orphaned pins with no corresponding database record.
  • The global rate limiter does not prevent this because it limits request count, not concurrent body size.

Fix

Replace in-memory buffering with disk-based streaming:

```typescript
import { createWriteStream, unlink } from "fs";
import { tmpdir } from "os";
import { join } from "path";

const upload = multer({
storage: multer.diskStorage({
destination: (_req, _file, cb) => cb(null, tmpdir()),
filename: (_req, file, cb) => cb(null, ${Date.now()}-${file.originalname}),
}),
limits: { fileSize: MAX_FILE_BYTES },
});
```

Stream the file from disk to Pinata using a ReadStream rather than loading the buffer into memory. Delete the temp file in a finally block after the upload completes or errors.

Files Affected

  • src/routes/ipfs.ts
  • src/services/ipfs.ts (uploadToIPFS — signature change from Buffer to file path / stream)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions