Multiple copies of image layers maintained in memory during image pull

[anakrish]

Currently during an image pull, multiple copies of each layer are maintained in memory before it is written out (possibly to secure storage):

1. Each layer's blob is first pulled into an in memory vector. https://github.com/confidential-containers/image-rs/blob/c98c4916cbe8d2c4f8933b343d20b7fdb338f50e/src/pull.rs#L91-L95
2. If the blob is encrypted, then it is decrypted to another in-memory vector. https://github.com/confidential-containers/image-rs/blob/c98c4916cbe8d2c4f8933b343d20b7fdb338f50e/src/pull.rs#L125-L127
3. Then if the layer is compressed, it is decompressed to another in memory vector. https://github.com/confidential-containers/image-rs/blob/c98c4916cbe8d2c4f8933b343d20b7fdb338f50e/src/pull.rs#L166-L168
4. The out vector which contains the layer's blob processed via the above steps is finally unpacked to destination folder: https://github.com/confidential-containers/image-rs/blob/c98c4916cbe8d2c4f8933b343d20b7fdb338f50e/src/pull.rs#L198

Pulling docker.io/library/python:latest using an image-rs test program on a linux machine shows the following memory consumption under heaptrack

peak heap memory consumption:
1.5GB after 11.733s
peak RSS (including heaptrack overhead):
888.8MB
total memory leaked:
50.9MB (10.6kB suppressed)

Given a kata-VM's default memory configuration of 2GB, 1.5GB memory consumption during image pull may cause slow performance and possibly OOM.

[anakrish]

#24 reduces memory usage while pulling encrypted images.

[dcmiddle]

Hi @anakrish Is this resolved by #96 ?
(similar comment in #67 )