Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-53431

cri-o cannot pull image with certain characteristics

XMLWordPrintable

    • Yes
    • False
    • Hide

      None

      Show
      None

      This is mostly a dump of my understanding from an incident where build farms running 4.18.5 cannot pull quay.io/coreos/tflint and jobs are failing as a result.

      Miloslav has done some great initial analysis which I'll copy out here:

       
      Reproduced, and confirming the guess that this if fixed in c/image commit 3161ab11ec8e10a5063be3df6305e25cfd045338 .
      @sascha et. al, I’m handing this off to the Node team for turning that into product versions / branch names, and backporting decisions.

      Reproducer (fails with Podman commit be41ee413115e014b839fbe9adc46529fa3c5cfc , succeeds with 6639c20278d1dbda08da0af364cad0241b5b6169 )

      • Create an image with the canonical empty layer, but in an uncompressed version, and not a "throwaway" one. I did skopeo copy --dest-decompress }}{{[docker://quay.io/coreos/tflint]{{ dir:t}} and manually edited t/manifest.json to remove the "throwaway" fields, but presumably for you this was an image on some registry, and for testing purposes, it can be created on a registry. Alternatively, this can be triggered with some other layer’s compressed/uncompressed versions, that would probably be easier, but this way I reproduce exactly the error message.
      • podman system reset -f
      • podman pull quay.io/coreos/tflint # populate metadata cache about the empty layer's compressed/uncompressed variants}}
      • (If not working with the empty layer, podman rmi … to delete the version matching the compressed digest from the storage)
      • skopeo --insecure-policy copy dir:t containers-storage:uncompressed # create an uncompressed version of the empty layer in local storage
      • podman pull pull quay.io/coreos/tflint # finds a match for the empty layer -> uncompressed, with incorrect bookkeeping}}

      If you want this fixed in CRI-O, please file a bug with the reproducer etc. so that this does not die in Slack.

      Essential components:

      • schema1 image
      • compressed layer reference does not match, but we have a BlobInfoCache entry for the compressed digest recording an uncompressed digest, and a local layer matching the uncompressed digest exists in c/storage
         

      Full Slack discussion at https://redhat-internal.slack.com/archives/CK1AE4ZCK/p1742222948447239

       

              rhn-support-minmli Min Li
              rhn-support-sdodson Scott Dodson
              Aditi Sahay Aditi Sahay
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

                Created:
                Updated:
                Resolved: