Merge zip archives

[emil-peters]

As I'm uploading a large amount of small files, the mount of workers created crashes the page.
For this reason and to limit the amount of workers I've been trying to chunk the number of files being uploaded each time.
But my issue now is that the merged zip archives contain to much.. I'm guessing this is related to some header in the zip?

Is there an easy way to remove this header or to merge zips archives?
I'll also post my recursive async solution to chucked zipping of a large amount of files

    let zipChunk = {};
    const chunkSize = 50;
    let filesZipped = 0;
    let tmpZippedData = new Uint8Array();

    const getNextChunk = async (startIndex) => {
      await Promise.all(
        files.map((file, idx) => {
          if (idx < startIndex || idx - startIndex >= chunkSize) {
            return file;
          }

          return file.arrayBuffer().then((buffer) => {
            zipChunk[file.name] = new Uint8Array(buffer);
          });
        })
      );
    };

    const zipNextChunk = async (startIndex) => {
      zipChunk = {};
      await getNextChunk(startIndex);
     
      zip(zipChunk, { consume: true, level: 6 }, (_, data) => {
        tmpZippedData = new Uint8Array([...tmpZippedData, ...data]);

        filesZipped += chunkSize;
        if (filesZipped >= files.length) {
          const file = new File([tmpZippedData], `${name}.zip`);
          callback(file);
        } else {
          zipNextChunk(filesZipped);
        }
      });
    };

    zipNextChunk(0);

Am I in the right direction? Or would there be a smarter solution? Or a better way to merge the zipped chunks?

[101arrowz]

First of all, it's very difficult to merge ZIP files efficiently. Unfortunately there's a footer for each ZIP file that is both required and impossible to merge with another footer, so what you're doing won't work. Even if it were possible, please note that the following line is probably going to cause a memory issue:

tmpZippedData = new Uint8Array([...tmpZippedData, ...data]);

The spread operator is efficient for copying arrays but is horrendous for this situation because it creates a standard JS array before converting to Uint8Array; since JS numbers take 32 bits at minimum in V8, you more than quadruple memory usage every time you do this, and its slow. If you want to concatenate Uint8Array:

const newZippedData = new Uint8Array(tmpZippedData.length + data.length);
newZippedData.set(tmpZippedData);
newZippedData.set(data, tmpZippedData.length);
tmpZippedData = newZippedData;

The above still won't work because again, you can't concatenate ZIP files. If the files are all tiny, just use the synchronous API to avoid creating so many workers. This shouldn't freeze the browser with sufficiently small files.

const toZip = {};
await Promise.all(files.map(async file => {
  toZip[file.name] = new Uint8Array(await file.arrayBuffer());
}));
const zipped = zipSync(toZip);

Small files also benefit less from compression, so you may consider using { level: 0 }. This will dramatically improve performance too.

If you experience freezing in browsers, let me know and I'll give a more robust solution.

[emil-peters]

Hi, thanks for your quick answer.
It's good to know that merging zipfiles the way I was doing it is not the way to go.
In our product, uploading 600+ small files (couple MB's each) is not rare therefore we were looking for a more robust solution to 600+ workers being spawned.

I took your advice and used the synchronous API to zip all files to a compression of { level: 0 }
The problem is that uploading 800+ MB's is not really desired, I was thus thinking if it would be smart to use asynchronous gzip to then gzip the zip file to get a better compression rating to not having to upload a large file.

    const toZip = {};
    await Promise.all(
      files.map(async (file) => {
        toZip[file.name] = new Uint8Array(await file.arrayBuffer());
      })
    );
    const res = zipSync(toZip, { consume: true, level: 0 });
    
   gzip(res, { consume: true }, (err, data) => {
     callback(data)
   }

As the async gzip of a single zip file would not cause the huge amount of workers to be spawned. Would you aggree with this solution?
Or would something more robust be out there?

Kind regards,
Emil

[101arrowz]

This is a valid solution for sure, as long as it's internal only you will be saving a decent amount of bandwidth this way. I think this is room for fflate's API to be optimized, but I might have to look into that at a later date.

[kajkal]

I also need to merge zip files on the browser side. I made a POC on the node.js side with the help of an additional library called adm-zip. If I understand this correctly it should also be feasible on the browser side - the only problem is reading the header and buffer with compressed data of a single zip entry (adm-zip is node.js exclusive).

import { writeFile } from 'node:fs/promises';
import * as fflate from 'fflate';
import AdmZip from 'adm-zip';

function mergeZips(zipsToMerge: Uint8Array[]) {
    let zip: fflate.Zip;

    const toBufferPromise = new Promise<Buffer>((resolve, reject) => {
        const chunks: Uint8Array[] = [];
        zip = new fflate.Zip((err, chunk, final) => {
            if (err) return reject(err);
            chunks.push(chunk);
            if (final) resolve(Buffer.concat(chunks));
        });
    });

    for (const zipToMerge of zipsToMerge) {
        const admZip = new AdmZip(Buffer.from(zipToMerge));
        const entries = admZip.getEntries();
        for (const entry of entries) {
            const file = new fflate.ZipPassThrough(entry.name);
            Object.assign(file, {
                crc: entry.header.crc,
                size: entry.header.size,
                compression: entry.header.method,
                flag: entry.header.flags,
                mtime: entry.header.time,
            });
            zip!.add(file);
            file.ondata(null, entry.getCompressedData(), true);
        }
    }

    zip!.end();
    return toBufferPromise;
}

const zip1 = fflate.zipSync({
    'file-1.json': fflate.strToU8(JSON.stringify({ some: 'content 1' })),
    'file-2.json': fflate.strToU8(JSON.stringify({ some: 'content 2' })),
});
const zip2 = fflate.zipSync({
    'file-3.txt': fflate.strToU8('content 3'),
});

const merged = await mergeZips([ zip1, zip2 ]);
await writeFile('./merged.zip', merged);

The code above is pretty fast - I was able to merge a dozen .zip files into one single .zip file in less than 6 seconds.
The resulting .zip file contained ~300k files with an uncompressed size of ~8GB (compressed they occupy ~600MB).

@101arrowz what do you think? Is there a chance that fflate would offer access to buffers with compressed data or to a buffer with a zip entry header? As fflate is tree-shakable a new function for more detailed header reading can be completely optional and will not affect the bundle size of those not interested in it :)

[101arrowz]

Sorry for the delayed response! I think having proper ZIP merging support is something that could be implemented by making the ZIP API more robust as a whole; right now there's just an in-memory API and a best-effort streaming API, but I believe an object-oriented way to access the metadata and file entries of a ZIP file would not be too difficult to implement. I'll see what I can do.

[101arrowz]

I don't think this is something I'll have time to work on for the time being, so I'll recommend using zip.js for now.

[VitthalGupta]

Python Implementation

https://github.com/VitthalGupta/zip-merge