ImageListPullJob with imagePullPolicy=Always can't pull latest tag image when the image has changed

[zj452008181]

what's wrong

I created a ImageListPullJob to pull images in some nodes,and set imagePullPolicy: Always to pull the latest tag whatever image exists in these nodes.
but unexpectedly,it's not working,when i build a new image and tag it harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe:latest,i push it to my harbor registry,repoDigests update to "sha256:d9d70b3d".the next day imagepulljob run again,imageid in all nodes don't update,then i clean all images in node01,it can pull the real latest iamges.

image inspect in node04

"status": {
"id": "sha256:d5b08854cde36b2531131773a36b6920ee650b3575dbe91982451a39760dca5e",
"repoTags": [
"harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe:latest"
],
"repoDigests": [
"harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe@sha256:41f571b072b5f0a5ee85b1393dbd95de6bc3cc13a5bc826a412f9a0f066e6c6e"
],
"size": "30127273",
"uid": null,
"username": "",
"spec": null,
"pinned": false
}

image inspect in node01

"status": {
"id": "sha256:91849a5248a24bf524879ddb34192ddd291dc547c229abc6364d3d25c0db7788",
"repoTags": [
"harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe:latest"
],
"repoDigests": [
"harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe@sha256:d9d70b3d97587d027ce12d773cca1ed66e6d16dd8a563a5b9c7c216d84d7dbda"
],
"size": "30127842",
"uid": null,
"username": "",
"spec": null,
"pinned": false
}

version

k8s:v1.26.15
containerd://1.7.13
kruise:v1.7.2

ImageListPullJob:

apiVersion: apps.kruise.io/v1alpha1
kind: ImageListPullJob
metadata:
  name: job-with-always
spec:
  images:
  - 'harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe:latest'
  imagePullPolicy: Always
  parallelism: 2      # [optional] the maximal number of Nodes that pull this image at the same time, defaults to 1
    #  pullSecrets:
    #   - idc-harbor
  selector:            # [optional] the names or label selector to assign Nodes (only one of them can be set)
    names:
    - gz-dr-node01
    - gz-dr-node02
    - gz-dr-node03
    - gz-dr-node04
    - gz-dr-node05
    - gz-dr-node06
    - gz-dr-node07
    - gz-dr-node08
    # matchLabels:
    #   env: prod
  completionPolicy:
    type: Never                  # [optional] defaults to Always
      #activeDeadlineSeconds: 1200   # [optional] no default, only work for Always type
      #ttlSecondsAfterFinished: 300  # [optional] no default, only work for Always type
  pullPolicy:                     # [optional] defaults to backoffLimit=3, timeoutSeconds=600
    backoffLimit: 3
    timeoutSeconds: 300
  pullSecrets:
  - idc-harbor

ImagePullJob:

Name:         job-with-always-drgcb
Namespace:    default
Labels:       controller-revision-hash=56f9cc997
Annotations:  <none>
API Version:  apps.kruise.io/v1alpha1
Kind:         ImagePullJob
Metadata:
  Creation Timestamp:  2024-11-04T01:30:17Z
  Finalizers:
    apps.kruise.io/deletion-protection
  Generate Name:  job-with-always-
  Generation:     1
  Owner References:
    API Version:           apps.kruise.io/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  ImageListPullJob
    Name:                  job-with-always
    UID:                   f7eca1db-1d82-4e3d-8067-d26a1b221b7d
  Resource Version:        119767682
  UID:                     9d07f677-8993-4c6e-bb64-f63bb1d90be1
Spec:
  Completion Policy:
    Type:             Never
  Image:              harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe:latest
  Image Pull Policy:  Always
  Parallelism:        2
  Pull Policy:
    Backoff Limit:    3
    Timeout Seconds:  300
  Pull Secrets:
    idc-harbor
  Selector:
    Names:
      gz-dr-node01
      gz-dr-node02
      gz-dr-node03
      gz-dr-node04
      gz-dr-node05
      gz-dr-node06
      gz-dr-node07
      gz-dr-node08
Status:
  Active:      0
  Desired:     8
  Failed:      0
  Message:     job is running, progress 100.0%
  Start Time:  2024-11-04T01:39:58Z
  Succeeded:   8
Events:        <none>

nodeimage:gz-dr-node01

harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe:
  pullSecrets:
    - name: idc-harbor-p5lzl
      namespace: kruise-daemon-config
  tags:
    - createdAt: '2024-11-04T01:39:58Z'
      ownerReferences:
        - apiVersion: apps.kruise.io/v1alpha1
          kind: ImagePullJob
          name: job-with-always-drgcb
          namespace: default
          uid: 9d07f677-8993-4c6e-bb64-f63bb1d90be1
      pullPolicy:
        activeDeadlineSeconds: 1800
        backoffLimit: 3
        timeoutSeconds: 300
        ttlSecondsAfterFinished: 86017
      tag: latest
status:
harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe:
  Tags:
    Completion Time:  2024-11-04T01:40:01Z
    Image ID:         harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe@sha256:91849a5248a24bf524879ddb34192ddd291dc547c229abc6364d3d25c0db7788
    Phase:            Succeeded
    Progress:         100
    Start Time:       2024-11-04T01:40:01Z
    Tag:              latest

nodeimage:gz-dr-node04

harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe:
  pullSecrets:
    - name: idc-harbor-p5lzl
      namespace: kruise-daemon-config
  tags:
    - createdAt: '2024-11-04T03:12:30Z'
      ownerReferences:
        - apiVersion: apps.kruise.io/v1alpha1
          kind: ImagePullJob
          name: job-with-always-drgcb
          namespace: default
          uid: 9d07f677-8993-4c6e-bb64-f63bb1d90be1
      pullPolicy:
        activeDeadlineSeconds: 1800
        backoffLimit: 3
        timeoutSeconds: 300
        ttlSecondsAfterFinished: 86319
      tag: latest
      version: 1
status:
harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe:
  Tags:
    Completion Time:  2024-11-04T03:12:35Z
    Image ID:         harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe@sha256:d5b08854cde36b2531131773a36b6920ee650b3575dbe91982451a39760dca5e
    Phase:            Succeeded
    Progress:         100
    Start Time:       2024-11-04T03:12:35Z
    Tag:              latest
    Version:          1

logs:

gz-dr-node04

\"harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe\":{\"tags\":[{\"tag\":\"latest\",\"phase\":\"Succeeded\",\"progress\":100,\"startTime\":\"2024-11-04T03:12:35Z\",\"completionTime\":\"2024-11-04T03:12:35Z\",\"version\":1,\"imageID\":\"harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe@sha256:d5b08854cde36b2531131773a36b6920ee650b3575dbe91982451a39760dca5e\"}]}

gz-dr-node01

\"harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe\":{\"tags\":[{\"tag\":\"latest\",\"phase\":\"Succeeded\",\"progress\":100,\"startTime\":\"2024-11-04T01:40:01Z\",\"completionTime\":\"2024-11-04T01:40:01Z\",\"imageID\":\"harbor.jusda.int/juslink-ms/juslink-4pl-spm-fe@sha256:91849a5248a24bf524879ddb34192ddd291dc547c229abc6364d3d25c0db7788\"}]}

[zmberg]

@zj452008181 Can you show the kruise-daemon logs ?

[zj452008181]

@zj452008181 Can you show the kruise-daemon logs ?

i chage log level to 5,it's too many
node01-daemon.log
node04-daemon.log

[zj452008181]

@zmberg Is there anything else I can provide？

[zmberg]

Can you communicate with we on Dingding Group?

[zj452008181]

Can you communicate with we on Dingding Group?

yes，how to join the group？

[hantmac]

Can you communicate with we on Dingding Group?

@zmberg Maybe we should talk here and leave every comment, because some other developer may have the same issue which they can search this resolved issue then.