Bug 1826559 - Podman builds bad container images: invalid tar header error
Summary: Podman builds bad container images: invalid tar header error
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: podman
Version: 33
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Lokesh Mandvekar
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-22 00:55 UTC by smurfendrek123
Modified: 2021-03-15 12:54 UTC (History)
12 users (show)

Fixed In Version: podman-3.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-15 12:50:23 UTC
Type: Bug


Attachments (Terms of Use)

Description smurfendrek123 2020-04-22 00:55:25 UTC
Description of problem:


Version-Release number of selected component (if applicable): 1.8.2


How reproducible:


Steps to Reproduce:
1. Build container with podman
2. Push it to a registry (in this case an azure container registry)
3. Try to use it elsewhere (for example in gke)

Actual results:
GKE gives me the error:

Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:b745c9620c42ed8b3d44112811e039e9e673072cef3bb5740bb1d560a40b1f8a: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount603661419: archive/tar: invalid tar header: unknown

Expected results:
I expect the container image to work


Additional info:
Works fine with moby

Possibly related to:
https://github.com/containers/image/issues/733

Comment 1 Daniel Walsh 2020-09-11 19:30:15 UTC
Could this be an issue with OCI Format?  If you build the image with --format docker, does it work properly?

Comment 2 smurfendrek123 2020-09-17 12:37:03 UTC
Hmm, i just tried this again with podman 2.0.6 on fedora 32, and couldn't reproduce this issue again with just regular podman, so I didn't try --format docker. I suppose this issue can be closed, I will open it or a new one if I encounter this issue again.

Comment 3 smurfendrek123 2020-09-17 13:05:12 UTC
Hmm, people in that github issue linked in this report are still saying that the problem still occurs on podman 2, so i'll set this back to assigned, but I can't seem to reproduce it personally.

Comment 4 Miloslav Trmač 2020-09-17 13:10:26 UTC
(Note that reproducing this from a _clean state_ is not transparently obvious, because it depends on what Podman/Skopeo/… ”already know” about the destination. I.e. a push from a fresh machine/container should always work, but a second push of the same content to the same registry can trigger the bug.)

Comment 5 smurfendrek123 2020-09-29 15:28:41 UTC
I managed to reproduce this again, and rebuilding & repushing with --format docker did seem to fix the issue. Thanks for the suggestion.

Comment 6 Andreas Karis 2020-12-13 10:52:52 UTC
I just hit this on RHEL 8.3 with:

buildah bud --format docker -t ${IMG} .
podman push ${IMG}

I build a container yesterday with these commands and successfully pulled if with kind.

Today, I repushed to my private registry, destroyed the kind cluster, respawned the kind cluster, and ran into this:
~~~
Events:
  Type     Reason   Age               From     Message
  ----     ------   ----              ----     -------
  Warning  Failed   16s               kubelet  Failed to pull image "kind:5000/sosreport-centos": rpc error: code = Unknown desc = failed to pull and unpack image "kind:5000/sosreport-centos:latest": failed to extract layer sha256:2653d992f4ef2bfd27f94db643815aa567240c37732cae1405ad1c1309ee9859: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount790305976: archive/tar: invalid tar header: unknown
  Normal   BackOff  15s               kubelet  Back-off pulling image "kind:5000/sosreport-centos"
  Warning  Failed   15s               kubelet  Error: ImagePullBackOff
  Normal   Pulling  3s (x2 over 17s)  kubelet  Pulling image "kind:5000/sosreport-centos"
  Warning  Failed   2s (x2 over 16s)  kubelet  Error: ErrImagePull
  Warning  Failed   2s                kubelet  Failed to pull image "kind:5000/sosreport-centos": rpc error: code = Unknown desc = failed to pull and unpack image "kind:5000/sosreport-centos:latest": failed to extract layer sha256:2653d992f4ef2bfd27f94db643815aa567240c37732cae1405ad1c1309ee9859: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount412063914: archive/tar: invalid tar header: unknown
~~~

Here are my environment specifics:
~~~
[root@kind ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.3 (Ootpa)
[root@kind ~]# rpm -qa | grep buildah
buildah-1.15.1-2.module+el8.3.0+8221+97165c3f.x86_64
[root@kind ~]# rpm -qa | grep podman
podman-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64
podman-catatonit-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64
~~~

Comment 7 Quentin Haas 2021-01-18 17:44:34 UTC
Seeing this in both EL8.3 and Fedora 33, especially when interacting with a docker registry hosted in GitLab.  Those aren't as easy to fully clear out by end-users since some images remain cached even after deletion via the GUI.

As a prior comment[1] suggests, this is not as straight forward to repeat since the state of prior images is important in triggering it.  I posted steps to trigger it in a comment over in another issue[2].

It goes something like this in Fedora 33 (details in the prior bug report):
```
# details about my environment
$ cat /etc/redhat-release
Fedora release 33 (Thirty Three)
$ singularity --version
singularity version 3.7.0-1.fc33
$ podman --version
podman version 2.2.1
$ buildah --version
buildah version 1.18.0 (image-spec 1.0.1-dev, runtime-spec 1.0.2-dev)
$ skopeo --version
skopeo version 1.2.0

# first create a simple, sample dockerfile
$ cat Dockerfile.1
FROM centos:7

RUN dd if=/dev/zero of=/opt/foo.bin bs=16M count=1

# start a clean registry
$ podman run --name reg -d --rm -p 127.0.0.1:5000:5000 registry:2.7

# build the image as OCI format
$ podman build -t 127.0.0.1:5000/test:oci -f Dockerfile.1 .
# build the image as docker format
$ BUILDAH_FORMAT=docker podman build -t 127.0.0.1:5000/test:docker -f Dockerfile.1 .
# push the docker image before the oci image
$ podman push --tls-verify=false 127.0.0.1:5000/test:docker
$ podman push --tls-verify=false 127.0.0.1:5000/test:oci
# the oci fails
$ singularity pull --nohttps test-oci.sif docker://127.0.0.1:5000/test:oci
...
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: read next entry: archive/tar: invalid tar header
# the docker succeeds
$ singularity pull --nohttps test-docker.sif docker://127.0.0.1:5000/test:docker


# now swap the order in which they are uploaded after clearing the registry
$ podman stop reg
$ singularity cache clean
$ podman run --name reg -d --rm -p 127.0.0.1:5000:5000 registry:2.7
$ podman push --tls-verify=false 127.0.0.1:5000/test:oci
$ podman push --tls-verify=false 127.0.0.1:5000/test:docker
# the oci pull now succeeds
$ singularity pull --nohttps test-oci.sif docker://127.0.0.1:5000/test:oci
# the docker pull now fails
$ singularity pull --nohttps test-docker.sif docker://127.0.0.1:5000/test:docker
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: read next entry: archive/tar: invalid tar header
```
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1826559#c4
[2]: https://github.com/containers/buildah/issues/1589

Comment 9 Daniel Walsh 2021-01-18 18:02:37 UTC
Fixed in podman 3.1 when it is released.

Comment 10 Daniel Walsh 2021-01-28 14:45:54 UTC
Actually this will be fixed in podman 3.0 rc2 which should be released tomorrow.

Comment 12 Quentin Haas 2021-03-11 11:53:24 UTC
Thanks for fixing this bug!  It will allow us to finally ditch docker without fear of breaking our images in our gitlab container registry...

I suspect RHEL8 won't be getting Podman 3.x in the official repos for a while, and the kubic[1] releases don't support ppc64le RHEL8.  Guess I could use mock to build podman 3.x using the spec in RHEL8 ppc64le, but I have a bad feeling that some of the build dependency packages listed aren't available in the ppc64le RHEL8 repos (which might be why kubic doesn't have them)...

[1]: https://podman.io/getting-started/installation#installing-development-versions-of-podman

Comment 13 Lokesh Mandvekar 2021-03-15 12:50:23 UTC
CLosing as 3.0 is already in stable.

Comment 14 Lokesh Mandvekar 2021-03-15 12:54:20 UTC
(In reply to Quentin Haas from comment #12)
> Thanks for fixing this bug!  It will allow us to finally ditch docker
> without fear of breaking our images in our gitlab container registry...
> 
> I suspect RHEL8 won't be getting Podman 3.x in the official repos for a
> while, and the kubic[1] releases don't support ppc64le RHEL8.  Guess I could
> use mock to build podman 3.x using the spec in RHEL8 ppc64le, but I have a
> bad feeling that some of the build dependency packages listed aren't
> available in the ppc64le RHEL8 repos (which might be why kubic doesn't have
> them)...
> 
> [1]:
> https://podman.io/getting-started/installation#installing-development-
> versions-of-podman

Quentin, I'll try enabling the ppc64le repos for centos stream, so let's see how it goes. Feel free to file an issue on podman github for it and tag me (github: @lsm5) there.


Note You need to log in before you can comment on or make changes to this bug.