Description of problem: Version-Release number of selected component (if applicable): 1.8.2 How reproducible: Steps to Reproduce: 1. Build container with podman 2. Push it to a registry (in this case an azure container registry) 3. Try to use it elsewhere (for example in gke) Actual results: GKE gives me the error: Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:b745c9620c42ed8b3d44112811e039e9e673072cef3bb5740bb1d560a40b1f8a: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount603661419: archive/tar: invalid tar header: unknown Expected results: I expect the container image to work Additional info: Works fine with moby Possibly related to: https://github.com/containers/image/issues/733
Could this be an issue with OCI Format? If you build the image with --format docker, does it work properly?
Hmm, i just tried this again with podman 2.0.6 on fedora 32, and couldn't reproduce this issue again with just regular podman, so I didn't try --format docker. I suppose this issue can be closed, I will open it or a new one if I encounter this issue again.
Hmm, people in that github issue linked in this report are still saying that the problem still occurs on podman 2, so i'll set this back to assigned, but I can't seem to reproduce it personally.
(Note that reproducing this from a _clean state_ is not transparently obvious, because it depends on what Podman/Skopeo/… ”already know” about the destination. I.e. a push from a fresh machine/container should always work, but a second push of the same content to the same registry can trigger the bug.)
I managed to reproduce this again, and rebuilding & repushing with --format docker did seem to fix the issue. Thanks for the suggestion.
I just hit this on RHEL 8.3 with: buildah bud --format docker -t ${IMG} . podman push ${IMG} I build a container yesterday with these commands and successfully pulled if with kind. Today, I repushed to my private registry, destroyed the kind cluster, respawned the kind cluster, and ran into this: ~~~ Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Failed 16s kubelet Failed to pull image "kind:5000/sosreport-centos": rpc error: code = Unknown desc = failed to pull and unpack image "kind:5000/sosreport-centos:latest": failed to extract layer sha256:2653d992f4ef2bfd27f94db643815aa567240c37732cae1405ad1c1309ee9859: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount790305976: archive/tar: invalid tar header: unknown Normal BackOff 15s kubelet Back-off pulling image "kind:5000/sosreport-centos" Warning Failed 15s kubelet Error: ImagePullBackOff Normal Pulling 3s (x2 over 17s) kubelet Pulling image "kind:5000/sosreport-centos" Warning Failed 2s (x2 over 16s) kubelet Error: ErrImagePull Warning Failed 2s kubelet Failed to pull image "kind:5000/sosreport-centos": rpc error: code = Unknown desc = failed to pull and unpack image "kind:5000/sosreport-centos:latest": failed to extract layer sha256:2653d992f4ef2bfd27f94db643815aa567240c37732cae1405ad1c1309ee9859: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount412063914: archive/tar: invalid tar header: unknown ~~~ Here are my environment specifics: ~~~ [root@kind ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.3 (Ootpa) [root@kind ~]# rpm -qa | grep buildah buildah-1.15.1-2.module+el8.3.0+8221+97165c3f.x86_64 [root@kind ~]# rpm -qa | grep podman podman-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64 podman-catatonit-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64 ~~~
Seeing this in both EL8.3 and Fedora 33, especially when interacting with a docker registry hosted in GitLab. Those aren't as easy to fully clear out by end-users since some images remain cached even after deletion via the GUI. As a prior comment[1] suggests, this is not as straight forward to repeat since the state of prior images is important in triggering it. I posted steps to trigger it in a comment over in another issue[2]. It goes something like this in Fedora 33 (details in the prior bug report): ``` # details about my environment $ cat /etc/redhat-release Fedora release 33 (Thirty Three) $ singularity --version singularity version 3.7.0-1.fc33 $ podman --version podman version 2.2.1 $ buildah --version buildah version 1.18.0 (image-spec 1.0.1-dev, runtime-spec 1.0.2-dev) $ skopeo --version skopeo version 1.2.0 # first create a simple, sample dockerfile $ cat Dockerfile.1 FROM centos:7 RUN dd if=/dev/zero of=/opt/foo.bin bs=16M count=1 # start a clean registry $ podman run --name reg -d --rm -p 127.0.0.1:5000:5000 registry:2.7 # build the image as OCI format $ podman build -t 127.0.0.1:5000/test:oci -f Dockerfile.1 . # build the image as docker format $ BUILDAH_FORMAT=docker podman build -t 127.0.0.1:5000/test:docker -f Dockerfile.1 . # push the docker image before the oci image $ podman push --tls-verify=false 127.0.0.1:5000/test:docker $ podman push --tls-verify=false 127.0.0.1:5000/test:oci # the oci fails $ singularity pull --nohttps test-oci.sif docker://127.0.0.1:5000/test:oci ... FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: read next entry: archive/tar: invalid tar header # the docker succeeds $ singularity pull --nohttps test-docker.sif docker://127.0.0.1:5000/test:docker # now swap the order in which they are uploaded after clearing the registry $ podman stop reg $ singularity cache clean $ podman run --name reg -d --rm -p 127.0.0.1:5000:5000 registry:2.7 $ podman push --tls-verify=false 127.0.0.1:5000/test:oci $ podman push --tls-verify=false 127.0.0.1:5000/test:docker # the oci pull now succeeds $ singularity pull --nohttps test-oci.sif docker://127.0.0.1:5000/test:oci # the docker pull now fails $ singularity pull --nohttps test-docker.sif docker://127.0.0.1:5000/test:docker FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: read next entry: archive/tar: invalid tar header ``` [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1826559#c4 [2]: https://github.com/containers/buildah/issues/1589
Fixed in podman 3.1 when it is released.
Actually this will be fixed in podman 3.0 rc2 which should be released tomorrow.
Thanks for fixing this bug! It will allow us to finally ditch docker without fear of breaking our images in our gitlab container registry... I suspect RHEL8 won't be getting Podman 3.x in the official repos for a while, and the kubic[1] releases don't support ppc64le RHEL8. Guess I could use mock to build podman 3.x using the spec in RHEL8 ppc64le, but I have a bad feeling that some of the build dependency packages listed aren't available in the ppc64le RHEL8 repos (which might be why kubic doesn't have them)... [1]: https://podman.io/getting-started/installation#installing-development-versions-of-podman
CLosing as 3.0 is already in stable.
(In reply to Quentin Haas from comment #12) > Thanks for fixing this bug! It will allow us to finally ditch docker > without fear of breaking our images in our gitlab container registry... > > I suspect RHEL8 won't be getting Podman 3.x in the official repos for a > while, and the kubic[1] releases don't support ppc64le RHEL8. Guess I could > use mock to build podman 3.x using the spec in RHEL8 ppc64le, but I have a > bad feeling that some of the build dependency packages listed aren't > available in the ppc64le RHEL8 repos (which might be why kubic doesn't have > them)... > > [1]: > https://podman.io/getting-started/installation#installing-development- > versions-of-podman Quentin, I'll try enabling the ppc64le repos for centos stream, so let's see how it goes. Feel free to file an issue on podman github for it and tag me (github: @lsm5) there.