From 9033b25dbb5961ab383297c95288113ab87726fd Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Wed, 15 Oct 2025 09:16:30 -0400 Subject: [PATCH 1/2] Initial sketch at design --- doc/design/use-oci-1.md | 68 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 doc/design/use-oci-1.md diff --git a/doc/design/use-oci-1.md b/doc/design/use-oci-1.md new file mode 100644 index 00000000..ef324aff --- /dev/null +++ b/doc/design/use-oci-1.md @@ -0,0 +1,68 @@ +Plan to refactor codebase on how we approach creation of singularity/apptainer containers. + +We want to not create them directly from docker images, but rather first rely +on functionality in https://github.com/datalad/datalad-container/pull/277 +(skopeo branch of the https://github.com/yarikoptic/datalad-container/ fork) to +initiate OCI container locally using `datalad containers-add oci:docker://...` +under `images-oci/` subdataset, under similar path (e.g. +repronim/repronim-reproin--0.13.1.oci for +images/repronim/repronim-reproin--0.13.1.sing in this one), registering it to +be ran with `{img_dspath}/scripts/oci_cmd run` which we are to provide as well. +E.g. + datalad containers-add --url oci:docker://bids/aa:v0.2.0 -i bids/bids-aa--0.2.0.oci bids-aa + +under images-oci// subdataset. + +While generating such OCI image we need to ensure that either all produced +files are under annex with URL or directly in git (if text files), e.g. + + git annex find --not --in datalad --and --not --in web bids/bids-aa--0.2.0.oci + +(could be under web directly or via datalad downloader!) + +`scripts/oci_cmd` could be simple for now: + + #!/bin/bash + + apptainer "$@" + +Then, after generation of OCI image, we would need to produce singularity SIF file using +(assuming that {image} would be the replacement with portion of path to image file like repronim/repronim-reproin--0.13.1) + + datalad run -m "Build SIF image for {image}.sif" --output images/{image}.sif scripts/oci_cmd build images/{image}.sif images-oci/{image}.oci/ + + +After all that done and works, we would need to have a migration +functionality which would produce .sif to replace all images for which we had Singularity* files but without custom commands, rather just basic wrappers. Full list could be obtained using + + git grep -l 'Automagically prepared' images + +and files would look like + + ❯ head images/bids/Singularity.bids-aa--0.2.0 + # + # Automagically prepared for ReproNim/containers distribution. + # See http://github.com/ReproNim/containers for more info + # + Bootstrap: docker + From: bids/aa:v0.2.0 + +so the goal would be to produce OCI image taking that "From:" as pointing to docker hub, in the above example (ran under images-oci/ subdataset). So the command to "containers-add" would be similar to above example: + + datalad containers-add --url oci:docker://bids/aa:v0.2.0 -i bids/bids-aa--0.2.0.oci bids-aa + +and then verifying that all annex files are available from URLs: + + git annex find --not --in datalad --and --not --in web bids/bids-aa--0.2.0.oci + +should come out empty. (so we need a generic helper function to be used here to reuse) + +Original images, and corresponding recipes, like in this case +images/bids/Singularity.bids-aa--0.2.0 where "From:" was found, and the corresponding image images/bids/bids-aa--0.2.0.sing should be "git rm"ed and committed with informative message. Path to the image within .datalad.config should be replaced to point to .sif instead of original .sing version. + +While developing, try migration first on some simpler cases like + + images/bids/bids-validator--1.2.3.sing + images/bids/bids-rshrf--1.0.0.sing + +For migration, add an option to skip failing, and we would need some log file listing those which failed to convert. From 0528f55569f8a72cb25c58047aebb4a8db616334 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Mon, 20 Oct 2025 08:39:16 -0400 Subject: [PATCH 2/2] syntax fixes from gemini Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- doc/design/use-oci-1.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/design/use-oci-1.md b/doc/design/use-oci-1.md index ef324aff..0cfa30ed 100644 --- a/doc/design/use-oci-1.md +++ b/doc/design/use-oci-1.md @@ -5,7 +5,7 @@ on functionality in https://github.com/datalad/datalad-container/pull/277 (skopeo branch of the https://github.com/yarikoptic/datalad-container/ fork) to initiate OCI container locally using `datalad containers-add oci:docker://...` under `images-oci/` subdataset, under similar path (e.g. -repronim/repronim-reproin--0.13.1.oci for +repronim/repronim-reproin--0.13.1.oci for images/repronim/repronim-reproin--0.13.1.sing in this one), registering it to be ran with `{img_dspath}/scripts/oci_cmd run` which we are to provide as well. E.g. @@ -55,10 +55,10 @@ and then verifying that all annex files are available from URLs: git annex find --not --in datalad --and --not --in web bids/bids-aa--0.2.0.oci -should come out empty. (so we need a generic helper function to be used here to reuse) +should come out empty. (so we need a generic helper function to be used here to reuse) Original images, and corresponding recipes, like in this case -images/bids/Singularity.bids-aa--0.2.0 where "From:" was found, and the corresponding image images/bids/bids-aa--0.2.0.sing should be "git rm"ed and committed with informative message. Path to the image within .datalad.config should be replaced to point to .sif instead of original .sing version. +images/bids/Singularity.bids-aa--0.2.0 where "From:" was found, and the corresponding image images/bids/bids-aa--0.2.0.sing should be "git rm"ed and committed with an informative message. Path to the image within .datalad.config should be replaced to point to .sif instead of original .sing version. While developing, try migration first on some simpler cases like