RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1853326 - podman segfault upon doing buildah operations due to error handling message
Summary: podman segfault upon doing buildah operations due to error handling message
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: podman
Version: 7.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 7.9
Assignee: Jindrich Novy
QA Contact: Martin Jenner
URL:
Whiteboard:
Depends On:
Blocks: 1186913 1807784
TreeView+ depends on / blocked
 
Reported: 2020-07-02 12:13 UTC by Robb Manes
Modified: 2023-12-15 18:22 UTC (History)
12 users (show)

Fixed In Version: podman-1.6.4-19.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-30 07:47:04 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Patch backpoerted to 7.9 by Valentin Rothberg (128.61 KB, application/mbox)
2020-07-29 13:51 UTC, Jindrich Novy
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github containers/podman/commit/76f42bd21c851aa1ad9c9e6f20d05890787d5276 0 None None None 2020-10-05 18:47:26 UTC
Red Hat Product Errata RHBA-2020:4089 0 None None None 2020-09-30 07:47:19 UTC

Description Robb Manes 2020-07-02 12:13:05 UTC
Description of problem:
Originally, we had an issue where podman segfaulted upon attempting to do a build from a remote image from a JFrog Artifactory repository like so:

# podman build --creds user:pass --tag me:test -f ./Dockerfile
STEP 1: FROM mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x55e0f9f69f18]

Checking the goroutine at fault, I found this to be fixed in https://github.com/containers/buildah/pull/1960/commits/62b7fcef3d049191f16d55c61e169c799ebdb221, so to be sure of the resolution, I took the customer's version of podman, added the vendored commit, and had it rebuilt to ensure that was the fix.  Although it definitely fixed the segfault, I was expecting a MIME-type mismatch or something similar judging from the segfault, and as I had no core, I couldn't actually check the Content-Types sent.  When they retested, however, the error message this commit introduced produced the following:

# podman build --creds user:pass --tag me:test -f ./Dockerfile
STEP 1: FROM mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
Error: error creating build container: error preparing image configuration: error converting image "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]" from "application/vnd.docker.distribution.manifest.list.v2+json" to "application/vnd.docker.distribution.manifest.v2+json": Conversion of image manifest from application/vnd.docker.distribution.manifest.v2+json to application/vnd.docker.distribution.manifest.v2+json is not implemented

I added additional debugging in the form of https://github.com/containers/image/pull/201/commits/36b3e49b50ae9c28676fc26822b13b9ecfa39842 which shows that the manifest types in use are in fact as described; I can attach these if you'd like.

Note this only happens on build attempts with this image so far as I can tell.  Pulling or inspecting the image works fine, and if you delete it locally and pull it first, then attempt to build, it still fails:

# skopeo inspect --creds user:pass mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
{
    "Name": "mydocker-registry.artifactory.example.com/ubi7/ubi",
    "Digest": "sha256:[SOMEHASH]",
    "RepoTags": [
        "7.6",
        "7.6-123",
        "7.6-177",
        "7.6-177.1561066496",
        "7.6-177.1561619828",
        "7.6-239",
        "7.6-73",
        "7.7",
        "7.7-140",
        "7.7-178",
        "7.7-214",
        "7.7-214.1575996166",
        "7.7-214.1580117713",
        "7.7-310",
        "7.7-310-source",
        "7.7-358",
        "7.7-358-source",
        "7.7-99",
        "7.8",
        "7.8-255",
        "7.8-255-source",
        "7.8-311",
        "7.8-311-source",
        "7.8-345",
        "7.8-345-source",
        "latest"
    ],
    "Created": "2020-05-11T17:22:52.084344Z",
    "DockerVersion": "1.13.1",
    "Labels": {
        "architecture": "x86_64",
        "build-date": "2020-05-11T17:22:15.219602",
        "com.redhat.build-host": "cpt-1008.osbs.prod.upshift.rdu2.redhat.com",
        "com.redhat.component": "ubi7-container",
        "com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
        "description": "The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
        "distribution-scope": "public",
        "io.k8s.description": "The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
        "io.k8s.display-name": "Red Hat Universal Base Image 7",
        "io.openshift.tags": "base rhel7",
        "name": "ubi7",
        "release": "311",
        "summary": "Provides the latest release of the Red Hat Universal Base Image 7.",
        "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7/images/7.8-311",
        "vcs-ref": "c8670b8ad70b591aaaafec3922308cb8e013a354",
        "vcs-type": "git",
        "vendor": "Red Hat, Inc.",
        "version": "7.8"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:[SOMEHASH]"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "container=oci"
    ]
} 

# podman rmi -f -a
Untagged: mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
Deleted: d36cb7ab60042e6687e221c9bfdc4b0c674e7753cff56f71bc3bd66e957598cc

# podman pull --creds user:pass mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
Trying to pull mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]...
Getting image source signatures
Copying blob 82a8f4ea76cb done  
Copying blob a3ac36470b00 done  
Copying config d36cb7ab60 done  
Writing manifest to image destination
Storing signatures
d36cb7ab60042e6687e221c9bfdc4b0c674e7753cff56f71bc3bd66e957598cc

# podman build --creds user:pass --tag me:test -f ./Dockerfile
STEP 1: FROM mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
Error: error creating build container: error preparing image configuration: error converting image "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]" from "application/vnd.docker.distribution.manifest.list.v2+json" to "application/vnd.docker.distribution.manifest.v2+json": Conversion of image manifest from application/vnd.docker.distribution.manifest.v2+json to application/vnd.docker.distribution.manifest.v2+json is not implemented

I am at the moment at a loss how we are failing from "Conversion of image manifest from application/vnd.docker.distribution.manifest.v2+json to application/vnd.docker.distribution.manifest.v2+json is not implemented" but continuing to look into it.  I am assuming that, in addition to this patch, we added some other facility in a later version that isn't present and subsequently the one patch isn't enough.

Version-Release number of selected component (if applicable):
podman-1.6.4-18.el7_8 with above error-handling patch cherry-picked on top.

How reproducible:
Every time in customer environment

Steps to Reproduce:
Attempt to build 

Actual results:
Segfaults/Errors on building images

Expected results:
Image should build

Additional info:
I will have them test just a generally later version of podman, and am requesting a copy of the image in the interim.

Comment 2 Daniel Walsh 2020-07-02 12:52:38 UTC
If you add the . after the -f does it work?


# podman build --creds user:pass --tag me:test -f ./Dockerfile .

What version of Podman are you using.

This might not get fixed because RHEL7 is no longer getting updates other then CVEs and Severe Bugs, if
this can easily be worked around, it will not be considered severe.

This should work fine in RHEL8 podman 1.9.*

Comment 3 Tom Sweeney 2020-07-02 17:54:05 UTC
Adding a period at the end as Dan suggested should work.  This was fixed in a later release, and certainly in RHEL 8/Podman v1.9+

Comment 4 Robb Manes 2020-07-06 14:12:29 UTC
(In reply to Daniel Walsh from comment #2)
> If you add the . after the -f does it work?

It does not, unfortunately.

# podman build --creds user:pass --tag me:test -f ./Dockerfile.proxysha .
STEP 1: FROM mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]
Error: error creating build container: error preparing image configuration: error converting image "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker-mydocker-registry.artifactory.example.com/ubi7/ubi@sha256:[SOMEHASH]" from "application/vnd.docker.distribution.manifest.list.v2+json" to "application/vnd.docker.distribution.manifest.v2+json": Conversion of image manifest from application/vnd.docker.distribution.manifest.v2+json to application/vnd.docker.distribution.manifest.v2+json is not implemented

> 
> What version of Podman are you using.

podman-1.6.4-18.el7_8 with above error-handling patch cherry-picked on top.

> 
> This might not get fixed because RHEL7 is no longer getting updates other
> then CVEs and Severe Bugs, if
> this can easily be worked around, it will not be considered severe.
> 
> This should work fine in RHEL8 podman 1.9.*

I can certainly ask if they can try a RHEL8 system.  It doesn't look like latest RHEL7 podman + my patch makes a difference.

Comment 5 Robb Manes 2020-07-08 19:21:06 UTC
Trying a RHEL8 system, using latest podman, works properly.  This appears just to be a problem on RHEL7 podman latest.

Adding a period does not seem to work.  I've asked for the image exported to see if I can duplicate the problem and try some commits that seem relevant.

We probably need another commit(s) than what I included to RHEL7 podman to resolve this, if you happen to know what those are off the top of the head it would be appreciated so we can see what needs to be backported.

Comment 6 Robb Manes 2020-07-13 16:14:48 UTC
I think I have muddied the waters a bit too much here after re-reading the bug, so I'd like to apologize.  Let me try again and be more clear.

If you build an image using `podman build` from certain images, in this case "registry.redhat.io/ubi7/ubi@sha256:bf8c66d697382ec1593a5f10db55c5c64e4de09046d21cd4c60066c5efb22412", it will segfault during unmarshalling.  This happens in RHEL7 and RHEL8 on podman latest for both; example using podman-1.6.4-12.module+el8.2.0+6669+dde598ec.x86_64:

# podman build -f ./Dockerfile .
STEP 1: FROM registry.redhat.io/ubi7/ubi@sha256:bf8c66d697382ec1593a5f10db55c5c64e4de09046d21cd4c60066c5efb22412
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x563c97a925a8]

Adding a period at the end does not make a difference in my testing.  This continues until v.1.7.0 where we introduce these two commits vendored into buildah:

https://github.com/containers/buildah/pull/1960/commits
https://github.com/containers/buildah/pull/1960/commits/62b7fcef3d049191f16d55c61e169c799ebdb221
https://github.com/containers/buildah/pull/1960/commits/e53efd6d1626cefb4b10f16718fa982544480e6e

If I cherry-pick just these from buildah into the appropriate vendored location in podman 1.6.4 or 1.6.5, it resolves the problem and we can build fine.

I understand one option is to wait until RHEL8 is at v1.7.0 since it introduces the commits; since there is not a workaround (that I can find or know of) for RHEL7, can we consider backporting this into the necessary Z-streams for both RHEL7 and RHEL8?  I don't know how we handle vendored commits into z-streams, if we allow cherry-picks into vendor/ or stick to just rebasing modules.  Thanks!

Comment 7 Daniel Walsh 2020-07-14 19:27:00 UTC
This will be fixed in RHEL8.2.1 which should be released soon.

As far as fixing it in rhel7.8 that is a little more difficult, and has to get PM approval, since we have no plans currently to udpate RHEL7 boxes.

Comment 11 Jindrich Novy 2020-07-29 13:51:47 UTC
Created attachment 1702811 [details]
Patch backpoerted to 7.9 by Valentin Rothberg

Comment 14 Joy Pu 2020-08-03 14:37:14 UTC
Can reproduce with podman-1.6.4-18.el7_8.x86_64 and test with podman-1.6.4-20.el7_9.x86_64. The command can finished as expect. So set this to verified. Details:

#  podman build  --tag me:test -f ./Dockerfile .
STEP 1: FROM registry-proxy.engineering.redhat.com/rh-osbs/ubi7@sha256:a8884b9b7039bae8a11e13b80c905bc7448aa654b9cbf7c7e1ea7763bf730769
STEP 2: RUN ls
bin   dev  home  lib64	mnt  proc  run	 srv  tmp  var
boot  etc  lib	 media	opt  root  sbin  sys  usr
STEP 3: COMMIT me:test
ea5233d563b356a192737819a6cd4bb36b144b8fb5805eb6ec2969a161bf8312
ea5233d563b356a192737819a6cd4bb36b144b8fb5805eb6ec2969a161bf8312
# podman  images
REPOSITORY                                           TAG      IMAGE ID       CREATED              SIZE
localhost/me                                         test     ea5233d563b3   About a minute ago   215 MB
registry-proxy.engineering.redhat.com/rh-osbs/ubi7   <none>   d36cb7ab6004   2 months ago         215 MB
docker.io/library/busybox                            latest   19485c79a9bb   11 months ago        1.44 MB

Comment 16 errata-xmlrpc 2020-09-30 07:47:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (podman bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4089


Note You need to log in before you can comment on or make changes to this bug.