Bug 1818694

Summary:	Golang panic when pushing image to a scaled image-registry
Product:	Red Hat Enterprise Linux 8	Reporter:	Robert Sandu <rsandu>
Component:	buildah	Assignee:	Jindrich Novy <jnovy>
Status:	CLOSED ERRATA	QA Contact:	atomic-bugs <atomic-bugs>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	8.1	CC:	aos-bugs, dornelas, dwalsh, gvillani, jnovy, mitr, nalin, obulatov, pazogni, tsweeney, ypu
Target Milestone:	rc	Keywords:	Reopened
Target Release:	8.3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	buildah-1.15.0 or newer	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-11-04 03:05:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1186913, 1804543

Description Robert Sandu 2020-03-30 06:33:38 UTC

Description of problem: after scaling the image-registry to `--replicas=2` new digests fails to push with a golang `panic: d.nx != 0`.

Version-Release number of selected component (if applicable): 4.3.8

How reproducible: attempted to replicate the issue described on two separate AWS clusters (with 2 registry replicas), and both attempts completed successfully.

Steps to Reproduce:
1. Scale image-registry to `--replicas=2`.
2. Push a new image to the image-registry.

Actual results: image push fails with golang `panic: d.nx != 0`

Expected results: image push to succeed.

Additional info:

- Scaling the image registry to `--replicas=1` avoids the issue.
- HTTP proxy used in the cluster.
- AWS cluster with persistent S3 storage set for the image registry.

Comment 6 Tom Sweeney 2020-03-30 21:51:41 UTC

Robert,

I'm not familiar with the --replicas option, I think that's something in OpenShift.  From what I can see in the stack trace, there's no evidence of any Buildah bits in there.  Can you tell me how you did the push and why you think this is a Buildah issue?

Nalin any thoughts?

Comment 8 Tom Sweeney 2020-03-31 16:03:11 UTC

Miloslav, do you have any further insights?

Comment 14 Robert Sandu 2020-04-06 08:08:29 UTC

Hi Oleg.

The issue has been confirmed as resolved by the customer, after setting up the `httpSecret` to a random string.

Closing the bug. Thank you for your help.

Comment 16 Daniel Walsh 2020-06-03 14:59:16 UTC

Panic fix is in current buildah-1.14.9

Comment 17 Tom Sweeney 2020-06-03 17:42:39 UTC

Assigning to Jindrich in case there are any further packaging needs.

Comment 18 Miloslav Trmač 2020-06-03 18:41:27 UTC

AFAICS v1.14.9 (f3e0e16690ac1370953bb76516053acf399776a4) does not contain the fix; c/image 5.4.4 is required.

Comment 19 Tom Sweeney 2020-06-03 19:16:13 UTC

Thanks Miloslav, then that will be in Podman v2.0 and RHEL 8.3.

Comment 29 Joy Pu 2020-08-03 06:48:24 UTC

Checked the source code of podman with podman-2.0.3-1.module+el8.3.0+7505+fe51f0c6.src.rpm, and the pullreq already included, So set this to verified. Details:
# find . -name "upload_reader.go"
./vendor/github.com/containers/image/v5/internal/uploadreader/upload_reader.go

Comment 34 errata-xmlrpc 2020-11-04 03:05:13 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4694