Design (+implementation) to switch to support/use OCI containers to produce singularity images

[yarikoptic]

It should fix

• no runscript defined for this container #137

and it should open paths for use of OCI images directly using podman/docker while being able to access original layers from the corresponding hubs (ATM docker support in datalad-container is not up to par with that ... datalad/datalad-container#277 is needed, seems to work nicely)

[gemini-code-assist]

Summary of Changes

Hello @yarikoptic, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a design and initial implementation plan to modernize the container creation process by shifting from direct Docker image conversion to a more flexible OCI container-based workflow. The change aims to leverage OCI images directly, facilitating better integration with tools like Podman/Docker, and includes a detailed strategy for migrating existing Singularity images to the new format. This refactoring is intended to fix issue #137 and open new avenues for container management.

Highlights

• New Container Workflow: Introduces a new workflow for creating Singularity/Apptainer containers, shifting from direct Docker image creation to an intermediate OCI container format for enhanced flexibility and integration.
• OCI Container Integration: Details the process of initiating local OCI containers using datalad containers-add under a new images-oci/ subdataset, leveraging functionality from a specific datalad-container PR.
• Migration Strategy: Outlines a comprehensive plan to migrate existing .sing images (identified by 'Automagically prepared' recipes) to the new .sif format, including steps for generating OCI/SIF images and cleaning up old files.
• Dependency on datalad-container: Highlights the reliance on the skopeo branch of datalad-container (PR #277) for the core OCI container functionality, which is essential for this new approach.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[bpinsard]

Sorry for the delay.
IIUC the plan is to:

• merge the datalad-containers PR
• create a repronim/containers-oci datasets that:
  • adds containers from dockerhub/other into oci-archives.
  • does not require storing the layers as it points to original registry through skopeo.
  • is that a standalone datalad-containers-operable dataset through docker/podman/.. adapters ?
• the current repronim/containers
• rebuild all containers from that OCI dataset, or only the ones with issues?
• what about Singularity recipes with custom commands?

From what I understand, it looks like a good plan that allow tracking provenance from external registries through layers SHA and manifests, but still generate a datalad-managed and separately-hosted SIF file.

[yarikoptic]

• merge the datalad-containers PR
• create a repronim/containers-oci datasets that:

just 1 dataset. I thought to create repronim/containers--images-oci since to provide images-oci subfolder git submoule to this one but on a second thought indeed having a self-sufficient repronim/containers-oci would be better.

* adds containers from dockerhub/other into oci-archives.

yes

• does not require storing the layers as it points to original registry through skopeo.

yes, does not require but we will store to provide "backup" (in case of hub or hub's organization going down/removed)

• is that a standalone datalad-containers-operable dataset through docker/podman/.. adapters ?

per above 2nd thought I think it might be valuable indeed!

• the current repronim/containers
• rebuild all containers from that OCI dataset, or only the ones with issues?

all for which there would be an OCI image. This way we could unify it all. If someone wants prior build, they would be able to check it out/use

• what about Singularity recipes with custom commands?

keep as is for now.