Digest did not match #127
Comments
|
Looks like #97 has the same symptom |
|
@MAHDTech Could you try by removing all |
|
Hey @nlewo, Yeah I've tried where the only args are the dest creds and still has the same error.
- id: container
name: Build container
run: >
devenv container
--registry docker://${{ env.REGISTRY }}/${{ env.PROJECT }}/
--copy-args="--dest-creds=${{ github.actor }}:${{ secrets.GITHUB_TOKEN }}"
copy ${{ matrix.container }}Results in; Copying blob sha256:fd8a0cc19f3537e316cce2fe05818f25ea0a89e64dffd5621aabc8503b36f894
time="2024-04-03T08:59:58Z" level=fatal msg="writing blob: Patch \"[https://ghcr.io/v2/***/containers/test/blobs/upload/e51cd0a5-7676-4187-b9af-70a4d451bd23\](https://ghcr.io/v2/***/containers/test/blobs/upload/e51cd0a5-7676-4187-b9af-70a4d451bd23/)": happened during read: Digest did not match, expected sha256:f0c636e0ef473fd1bb68a3bade4e24bdc202a0b86e05702bdad7c6f5453b26dd, got sha256:a908ca128957816d3b326cb489afe5db6d5bf1071113b0d983e3651d06b1efd3"
✔ Copying test container in 11.9s.
Error: × Failed to copy container |
|
Well, this kind of issue are currently pretty painful to debug because you need to compare how the layer hash is computed at build time to the layer hash computed at runtime. In nix2container, there is the option Are you able to reproduce locally? |
|
Root cause: NixOS/nix#10525 |
This commit is a workaround for NixOS/nix#10525 and nlewo/nix2container#127
|
@Atry Thank you and congrats for the debugging! (I would really like to write a tool generating the list of files (attribute + hash) at build time and runtime in order to compare them easily, but not enough time on my spare time.) |
|
I encountered the same error message when trying to use A small reproducible example, which uses forked devenv and nix2container to be able to pass git clone https://github.com/ento/devenv
cd devenv
git checkout container-digest-mismatch-repro
cd examples/container
direnv allow
# container that sets reproducible=false
devenv container run good
# container that sets reproducible=true
devenv container run badOutput of `devenv container run bad`
• Building bad container ...
• Using Cachix: devenv
warning: creating lock file '/examples/container/devenv.lock'
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
/nix/store/5vpy61qxm85izcx8bnhs24g5xlwzvrsg-image-bad.json
✔ Building bad container in 9.8s.
warning: creating lock file '/examples/container/devenv.lock'
• Running /nix/store/0rrxkx0lj6agbc72imvs5hkyyrv64xcy-copy-container /nix/store/5vpy61qxm85izcx8bnhs24g5xlwzvrsg-image-bad.json docker-daemon:
Copying container /nix/store/5vpy61qxm85izcx8bnhs24g5xlwzvrsg-image-bad.json to docker-daemon:bad:latest
Getting image source signatures
Copying blob db3b02af90da done |
Copying blob c99a1b2b3ae9 [=====================================>] 391.2MiB / 391.2MiB | 30.5 MiB/s
FATA[0001] writing blob: writing to temporary on-disk layer: happened during read: Digest did not match, expected sha256:c99a1b2b3ae9337c455c3ffe47eb8b54f5209b07e9b26261cd15a083bbcb755a, got sha256:656d7d142bca6908cb75c40e2f1150e3ae7855d28a4fb0a18268cc40a6f2e359
✔ Copying bad container in 2.7s.
Error: × Failed to copy containerOutput of `devenv contaienr run good`This doesn't actually succeed in running the image, but I'm guessing it's a separate issue.
$ devenv container run good
• Building good container ...
• Using Cachix: devenv
warning: creating lock file '/examples/container/devenv.lock'
/nix/store/7q0icf7bw0ngqfjmh5kxnnmgdb0zy8ag-image-good.json
✔ Building good container in 1.5s.
warning: creating lock file '/examples/container/devenv.lock'
• Running /nix/store/k4hl27rchhpkhrq76278kq3pphpgz158-copy-container /nix/store/7q0icf7bw0ngqfjmh5kxnnmgdb0zy8ag-image-good.json docker-daemon:
Copying container /nix/store/7q0icf7bw0ngqfjmh5kxnnmgdb0zy8ag-image-good.json to docker-daemon:good:latest
Getting image source signatures
Copying blob db3b02af90da done |
Copying blob c99a1b2b3ae9 done |
Copying config dd21b4c201 done |
Writing manifest to image destination
✔ Copying good container in 2.5s.
warning: creating lock file '/examples/container/devenv.lock'
mkdir: cannot create directory '/run': Permission denied
✔ Running good container in 1.3s.This gives me an image JSON file with layers pointing to tarballs, which I guess are build-time artifacts. I'm not sure how to get ahold of runtime blobs to compare against. |
|
Can you check what version of nix-daemon is running with |
|
Ohh. v2.22.0. $ nix store ping
warning: 'nix store ping' is a deprecated alias for 'nix store info'
Store URL: local
Version: 2.22.0
Trusted: 1This gave me hope because I'd rebooted my machine after posting my previous comment (so the daemon may not have been on the updated version then), but no, it still errors with the same message and hashes :/ $ devenv container run bad
...
FATA[0003] writing blob: writing to temporary on-disk layer: happened during read: Digest did not match, expected sha256:c99a1b2b3ae9337c455c3ffe47eb8b54f5209b07e9b26261cd15a083bbcb755a, got sha256:656d7d142bca6908cb75c40e2f1150e3ae7855d28a4fb0a18268cc40a6f2e359 |
|
I have not been able to reproduce: @ento I'm not sure, but it could be possible that the above nix issue poisoned your nix store. (Maybe you are reusing previously built store paths that doesn't have the correct hash because of the symlink issue.) @ento Could you clear your Nix store to be sure all store paths are built with the fixed Nix version? |
|
Is there a good way of clearning just the paths that'd need to be cleared? If not, I'll be happy to clear the entire store, but it'd be useful for others if there's a way to delete paths in a targeted fashion. I tried deleting paths selectively like this, and the issue persisted, although the digest value changed. The same after running image_path=/nix/store/bipgbyyrjhzl13lfh8jaajdm02kqsmhz-image-bad.json
deriver=$(nix-store --query --deriver $image_path)
nix-store -q --requisites --include-outputs $image_path > drv-closure.txt
nix-store -q --requisites --include-outputs $deriver >> drv-closure.txt
tac drv-closure.txt | xargs -L1 nix-store --deleteThis might be a red herring and might be expected, but digest values seem to change every time I run
diff between two image JSON files that were produced after running nix store gc
|
|
This is how I delete the cache: |
|
@ento I started working a pretty useful debug option: To debug your issue, you could checkout this MR: #135 The MR description and an added section provide some help on how to use this option. |
Hi, I'm using
devenvto build a container image withnix2containerand after the image is successfully created the following error message is shown when skopeo is uploading the image.Currently I'm using this setup
I've tried passing different copy flags to skopeo with the same result, including with and without
--preserve-digestsAnyone seen this before or have any tips where to look?
The text was updated successfully, but these errors were encountered: