Bug 2216700

Summary: podman images command fails while multiple builds are executed in parallel
Product: Red Hat Enterprise Linux 8 Reporter: Suhaas Bhat <subhat>
Component: podmanAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Yuhui Jiang <yujiang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.8CC: ajia, bbaude, dwalsh, jligon, jnovy, lsm5, mboddu, mheon, myamazak, pthomas, tsweeney, umohnani, vrothber
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: podman-4.6.1-2.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-14 15:29:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Suhaas Bhat 2023-06-22 09:50:34 UTC
Description of problem:
Podman throws below error if executed in a consecutive time of 1 sec ( although not important ) while multiple builds are getting executed in parallel

Error: retrieving label for image "e1d051d6dbc35587162d4782ddffdbfe34883a1516087f9e2516e4d24d74e451": you may need to remove the image to resolve the error: fallback error checking whether image is a manifest list: reading image "e1d051d6dbc35587162d4782ddffdbfe34883a1516087f9e2516e4d24d74e451": locating image with ID "e1d051d6dbc35587162d4782ddffdbfe34883a1516087f9e2516e4d24d74e451": image not known: reading image "e1d051d6dbc35587162d4782ddffdbfe34883a1516087f9e2516e4d24d74e451": locating image with ID "e1d051d6dbc35587162d4782ddffdbfe34883a1516087f9e2516e4d24d74e451": image not known

Version-Release number of selected component (if applicable):
podman-4.4.1-12.module+el8.8.0+18735+a32c1292.x86_64

Can be reproduced on RHEL 9 podman as well. 

How reproducible:
3 out of 5 tries

Steps to Reproduce:
Dockerfile :

FROM registry.access.redhat.com/ubi8/ubi
COPY foo /tmp

Command 1 : #for i in $(seq -w 00 07); do (echo $i > foo; podman build -t test$i .; podman rmi test$i)& done;

Command 2 : #while :; do echo start; podman images || break; echo end; echo; sleep 1; done

Error comes from :

containers-podman-aa5db47/pkg/domain/infra/abi/images_list.go :
-------------------------------------------------
   e := entities.ImageSummary{
                        ID:          img.ID(),
                        Created:     img.Created().Unix(),
                        Dangling: isDangling,
                        Digest:      string(img.Digest()),
                        RepoDigests: repoDigests,
                        History:     img.NamesHistory(),
                        Names:       img.Names(),
                        ReadOnly:    img.IsReadOnly(),
                        SharedSize:  0,
                        RepoTags:    img.Names(), // may include tags and digests
                }

               e.Labels, err = img.Labels(ctx)
                if err != nil {
                        return nil, fmt.Errorf("retrieving label for image %q: you may need to remove the image to resolve the error: %w", img.ID(), err)
                }
------------------------------------------------


Actual results:
#podman images command fails while executing multiple image builds

Expected results:
#podman images command should succeed every time unless there is a process kill while the build is in place.

Additional info:
https://github.com/containers/storage/pull/1113/commits/bb046fb8b4c729a099cd6ca5ce9a6d52fa717ee1   was added referencing the error.

Comment 1 Tom Sweeney 2023-06-22 22:57:40 UTC
Valentin, PTAL

Comment 2 Valentin Rothberg 2023-06-23 07:15:49 UTC
Thanks for pulling me in, Tom.

> Command 1 : #for i in $(seq -w 00 07); do (echo $i > foo; podman build -t test$i .; podman rmi test$i)& done;
> Command 2 : #while :; do echo start; podman images || break; echo end; echo; sleep 1; done

That is an interesting case.  Thank you for the nice reproducer.  I will prepare a fix for it.

Comment 3 Valentin Rothberg 2023-06-23 11:50:49 UTC
I opened https://github.com/containers/podman/pull/18980 along with a regression test.

Comment 4 Valentin Rothberg 2023-06-27 12:31:29 UTC
PR https://github.com/containers/podman/pull/18980 merged into the main branch.  It will make it into upcoming 4.6 release.

Assigning to Jindrich.

Comment 10 errata-xmlrpc 2023-11-14 15:29:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6939