Bug 1853326

Summary: podman segfault upon doing buildah operations due to error handling message
Product: Red Hat Enterprise Linux 7 Reporter: Robb Manes <rmanes>
Component: podmanAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Martin Jenner <mjenner>
Severity: high Docs Contact:
Priority: high    
Version: 7.8CC: bbaude, dornelas, dwalsh, hartsjc, imcleod, jligon, jnovy, lsm5, mheon, tsweeney, umohnani, ypu
Target Milestone: rcKeywords: Extras
Target Release: 7.9   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: podman-1.6.4-19.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-30 07:47:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1186913, 1807784    
Attachments:
Description Flags
Patch backpoerted to 7.9 by Valentin Rothberg none

Description Robb Manes 2020-07-02 12:13:05 UTC
Description of problem:
Originally, we had an issue where podman segfaulted upon attempting to do a build from a remote image from a JFrog Artifactory repository like so:

# podman build --creds user:pass --tag me:test -f ./Dockerfile
STEP 1: FROM mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x55e0f9f69f18]

Checking the goroutine at fault, I found this to be fixed in https://github.com/containers/buildah/pull/1960/commits/62b7fcef3d049191f16d55c61e169c799ebdb221, so to be sure of the resolution, I took the customer's version of podman, added the vendored commit, and had it rebuilt to ensure that was the fix.  Although it definitely fixed the segfault, I was expecting a MIME-type mismatch or something similar judging from the segfault, and as I had no core, I couldn't actually check the Content-Types sent.  When they retested, however, the error message this commit introduced produced the following:

# podman build --creds user:pass --tag me:test -f ./Dockerfile
STEP 1: FROM mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
Error: error creating build container: error preparing image configuration: error converting image "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]" from "application/vnd.docker.distribution.manifest.list.v2+json" to "application/vnd.docker.distribution.manifest.v2+json": Conversion of image manifest from application/vnd.docker.distribution.manifest.v2+json to application/vnd.docker.distribution.manifest.v2+json is not implemented

I added additional debugging in the form of https://github.com/containers/image/pull/201/commits/36b3e49b50ae9c28676fc26822b13b9ecfa39842 which shows that the manifest types in use are in fact as described; I can attach these if you'd like.

Note this only happens on build attempts with this image so far as I can tell.  Pulling or inspecting the image works fine, and if you delete it locally and pull it first, then attempt to build, it still fails:

# skopeo inspect --creds user:pass mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
{
    "Name": "mydocker-registry.artifactory.example.com/ubi7/ubi",
    "Digest": "sha256:[SOMEHASH]",
    "RepoTags": [
        "7.6",
        "7.6-123",
        "7.6-177",
        "7.6-177.1561066496",
        "7.6-177.1561619828",
        "7.6-239",
        "7.6-73",
        "7.7",
        "7.7-140",
        "7.7-178",
        "7.7-214",
        "7.7-214.1575996166",
        "7.7-214.1580117713",
        "7.7-310",
        "7.7-310-source",
        "7.7-358",
        "7.7-358-source",
        "7.7-99",
        "7.8",
        "7.8-255",
        "7.8-255-source",
        "7.8-311",
        "7.8-311-source",
        "7.8-345",
        "7.8-345-source",
        "latest"
    ],
    "Created": "2020-05-11T17:22:52.084344Z",
    "DockerVersion": "1.13.1",
    "Labels": {
        "architecture": "x86_64",
        "build-date": "2020-05-11T17:22:15.219602",
        "com.redhat.build-host": "cpt-1008.osbs.prod.upshift.rdu2.redhat.com",
        "com.redhat.component": "ubi7-container",
        "com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
        "description": "The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
        "distribution-scope": "public",
        "io.k8s.description": "The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
        "io.k8s.display-name": "Red Hat Universal Base Image 7",
        "io.openshift.tags": "base rhel7",
        "name": "ubi7",
        "release": "311",
        "summary": "Provides the latest release of the Red Hat Universal Base Image 7.",
        "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7/images/7.8-311",
        "vcs-ref": "c8670b8ad70b591aaaafec3922308cb8e013a354",
        "vcs-type": "git",
        "vendor": "Red Hat, Inc.",
        "version": "7.8"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:[SOMEHASH]"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "container=oci"
    ]
} 

# podman rmi -f -a
Untagged: mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
Deleted: d36cb7ab60042e6687e221c9bfdc4b0c674e7753cff56f71bc3bd66e957598cc

# podman pull --creds user:pass mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
Trying to pull mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]...
Getting image source signatures
Copying blob 82a8f4ea76cb done  
Copying blob a3ac36470b00 done  
Copying config d36cb7ab60 done  
Writing manifest to image destination
Storing signatures
d36cb7ab60042e6687e221c9bfdc4b0c674e7753cff56f71bc3bd66e957598cc

# podman build --creds user:pass --tag me:test -f ./Dockerfile
STEP 1: FROM mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
Error: error creating build container: error preparing image configuration: error converting image "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]" from "application/vnd.docker.distribution.manifest.list.v2+json" to "application/vnd.docker.distribution.manifest.v2+json": Conversion of image manifest from application/vnd.docker.distribution.manifest.v2+json to application/vnd.docker.distribution.manifest.v2+json is not implemented

I am at the moment at a loss how we are failing from "Conversion of image manifest from application/vnd.docker.distribution.manifest.v2+json to application/vnd.docker.distribution.manifest.v2+json is not implemented" but continuing to look into it.  I am assuming that, in addition to this patch, we added some other facility in a later version that isn't present and subsequently the one patch isn't enough.

Version-Release number of selected component (if applicable):
podman-1.6.4-18.el7_8 with above error-handling patch cherry-picked on top.

How reproducible:
Every time in customer environment

Steps to Reproduce:
Attempt to build 

Actual results:
Segfaults/Errors on building images

Expected results:
Image should build

Additional info:
I will have them test just a generally later version of podman, and am requesting a copy of the image in the interim.

Comment 2 Daniel Walsh 2020-07-02 12:52:38 UTC
If you add the . after the -f does it work?


# podman build --creds user:pass --tag me:test -f ./Dockerfile .

What version of Podman are you using.

This might not get fixed because RHEL7 is no longer getting updates other then CVEs and Severe Bugs, if
this can easily be worked around, it will not be considered severe.

This should work fine in RHEL8 podman 1.9.*

Comment 3 Tom Sweeney 2020-07-02 17:54:05 UTC
Adding a period at the end as Dan suggested should work.  This was fixed in a later release, and certainly in RHEL 8/Podman v1.9+

Comment 4 Robb Manes 2020-07-06 14:12:29 UTC
(In reply to Daniel Walsh from comment #2)
> If you add the . after the -f does it work?

It does not, unfortunately.

# podman build --creds user:pass --tag me:test -f ./Dockerfile.proxysha .
STEP 1: FROM mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
Error: error creating build container: error preparing image configuration: error converting image "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker-mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]" from "application/vnd.docker.distribution.manifest.list.v2+json" to "application/vnd.docker.distribution.manifest.v2+json": Conversion of image manifest from application/vnd.docker.distribution.manifest.v2+json to application/vnd.docker.distribution.manifest.v2+json is not implemented

> 
> What version of Podman are you using.

podman-1.6.4-18.el7_8 with above error-handling patch cherry-picked on top.

> 
> This might not get fixed because RHEL7 is no longer getting updates other
> then CVEs and Severe Bugs, if
> this can easily be worked around, it will not be considered severe.
> 
> This should work fine in RHEL8 podman 1.9.*

I can certainly ask if they can try a RHEL8 system.  It doesn't look like latest RHEL7 podman + my patch makes a difference.

Comment 5 Robb Manes 2020-07-08 19:21:06 UTC
Trying a RHEL8 system, using latest podman, works properly.  This appears just to be a problem on RHEL7 podman latest.

Adding a period does not seem to work.  I've asked for the image exported to see if I can duplicate the problem and try some commits that seem relevant.

We probably need another commit(s) than what I included to RHEL7 podman to resolve this, if you happen to know what those are off the top of the head it would be appreciated so we can see what needs to be backported.

Comment 6 Robb Manes 2020-07-13 16:14:48 UTC
I think I have muddied the waters a bit too much here after re-reading the bug, so I'd like to apologize.  Let me try again and be more clear.

If you build an image using `podman build` from certain images, in this case "registry.redhat.io/ubi7/ubi@sha256:bf8c66d697382ec1593a5f10db55c5c64e4de09046d21cd4c60066c5efb22412", it will segfault during unmarshalling.  This happens in RHEL7 and RHEL8 on podman latest for both; example using podman-1.6.4-12.module+el8.2.0+6669+dde598ec.x86_64:

# podman build -f ./Dockerfile .
STEP 1: FROM registry.redhat.io/ubi7/ubi@sha256:bf8c66d697382ec1593a5f10db55c5c64e4de09046d21cd4c60066c5efb22412
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x563c97a925a8]

Adding a period at the end does not make a difference in my testing.  This continues until v.1.7.0 where we introduce these two commits vendored into buildah:

https://github.com/containers/buildah/pull/1960/commits
https://github.com/containers/buildah/pull/1960/commits/62b7fcef3d049191f16d55c61e169c799ebdb221
https://github.com/containers/buildah/pull/1960/commits/e53efd6d1626cefb4b10f16718fa982544480e6e

If I cherry-pick just these from buildah into the appropriate vendored location in podman 1.6.4 or 1.6.5, it resolves the problem and we can build fine.

I understand one option is to wait until RHEL8 is at v1.7.0 since it introduces the commits; since there is not a workaround (that I can find or know of) for RHEL7, can we consider backporting this into the necessary Z-streams for both RHEL7 and RHEL8?  I don't know how we handle vendored commits into z-streams, if we allow cherry-picks into vendor/ or stick to just rebasing modules.  Thanks!

Comment 7 Daniel Walsh 2020-07-14 19:27:00 UTC
This will be fixed in RHEL8.2.1 which should be released soon.

As far as fixing it in rhel7.8 that is a little more difficult, and has to get PM approval, since we have no plans currently to udpate RHEL7 boxes.

Comment 11 Jindrich Novy 2020-07-29 13:51:47 UTC
Created attachment 1702811 [details]
Patch backpoerted to 7.9 by Valentin Rothberg

Comment 14 Joy Pu 2020-08-03 14:37:14 UTC
Can reproduce with podman-1.6.4-18.el7_8.x86_64 and test with podman-1.6.4-20.el7_9.x86_64. The command can finished as expect. So set this to verified. Details:

#  podman build  --tag me:test -f ./Dockerfile .
STEP 1: FROM registry-proxy.engineering.redhat.com/rh-osbs/ubi7@sha256:a8884b9b7039bae8a11e13b80c905bc7448aa654b9cbf7c7e1ea7763bf730769
STEP 2: RUN ls
bin   dev  home  lib64	mnt  proc  run	 srv  tmp  var
boot  etc  lib	 media	opt  root  sbin  sys  usr
STEP 3: COMMIT me:test
ea5233d563b356a192737819a6cd4bb36b144b8fb5805eb6ec2969a161bf8312
ea5233d563b356a192737819a6cd4bb36b144b8fb5805eb6ec2969a161bf8312
# podman  images
REPOSITORY                                           TAG      IMAGE ID       CREATED              SIZE
localhost/me                                         test     ea5233d563b3   About a minute ago   215 MB
registry-proxy.engineering.redhat.com/rh-osbs/ubi7   <none>   d36cb7ab6004   2 months ago         215 MB
docker.io/library/busybox                            latest   19485c79a9bb   11 months ago        1.44 MB

Comment 16 errata-xmlrpc 2020-09-30 07:47:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (podman bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4089