Bug 2000943

Summary:	podman auto update fails to login to registry after podman upgrade to 3.2
Product:	Red Hat Enterprise Linux 8	Reporter:	Rik Theys <rik.theys>
Component:	podman	Assignee:	Jindrich Novy <jnovy>
Status:	CLOSED ERRATA	QA Contact:	Joy Pu <ypu>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	8.4	CC:	bbaude, dwalsh, jligon, jnovy, lmiksik, lsm5, mheon, pthomas, smccarty, tsweeney, umohnani, vrothber, ypu
Target Milestone:	rc	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	podman-3.3.1-9.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2002591 2002670 2002721 (view as bug list)		Environment:
Last Closed:	2021-11-09 17:40:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2002591, 2002670, 2002721

Description Rik Theys 2021-09-03 10:56:32 UTC

Description of problem:
We have containers running with the following labels:

io.containers.autoupdate=image
io.containers.autoupdate.authfile=/etc/php-containers/auth.json

This makes sure the podman-auto-update service uses the correct credentials to access the registry.

This was working fine with podman 3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64, but has stopped working since we upgraded to 3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64

The log shows:

Aug 18 00:00:01 carbon.esat.kuleuven.be systemd[1]: Starting Podman auto-update service...
Aug 18 00:00:06 carbon.esat.kuleuven.be podman[631507]: Error: 73 errors occurred:
Aug 18 00:00:06 carbon.esat.kuleuven.be podman[631507]:         * error registry auto-updating container "01afda38c2439133e76c7effb059529de0a92494802c5021a7114b3bf4c886a2": image check for "gitlab-registry.esat.kuleuven.b>
Aug 18 00:00:06 carbon.esat.kuleuven.be podman[631507]:         * error registry auto-updating container "0e0b119652057cbc5f179a673c98192099c4b986413c1425fa8e904603574897": image check for "gitlab-registry.esat.kuleuven.b>

If we manually pull the image using:

podman pull --authfile /etc/php-containers/auth.json gitlab-registry.esat.kuleuven.be/sysgrp/php-container-image/c8-php74

it works fine. It seems podman is ignoring the authfile when trying to pull the new image.

Version-Release number of selected component (if applicable):
podman-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64

How reproducible:


Steps to Reproduce:
1. Label a container with the labels to have it automatically update from a repo that requires auth
2. Try to run it
3.

Actual results:
Errors about permissions for the registry

Expected results:
Working as before

Additional info:

Comment 1 Rik Theys 2021-09-03 10:58:18 UTC

It seems the error message I pasted was truncated. The full error line for the auto update is:

Sep 03 00:00:05 carbon.esat.kuleuven.be podman[2342926]:         * error registry auto-updating container "eeeb8e19a31f6c2d5b1fb420e6d59c96fc7ffb431e9fa5e1a7baa6ad9016747a": image check for "gitlab-registry.esat.kuleuven.be/sysgrp/php-container-image/c8-php74:latest" failed: Requesting bear token: invalid status code from registry 403 (Forbidden)

Comment 2 Valentin Rothberg 2021-09-03 11:13:52 UTC

Thanks for reaching, Rik. There is indeed a regression that has been fixed in the main branch [1] this week.

@Tom: Shall we backport? v3.3.1 only?


[1] https://github.com/containers/podman/issues/11171

Comment 3 Daniel Walsh 2021-09-07 17:37:04 UTC

I believe this should be fixed and back ported into v3.3.2

Comment 4 Tom Sweeney 2021-09-07 19:55:25 UTC

I think we ought to get an exception for this one and get it into v3.3.1-rhel so it can be delivered with RHEL 8.5 if approved.  It should also be backported into v3.3 so that it will be par of Podman v3.3.2+ that we deliver to RHEL 8.5.0.2 in 6 weeks or so.  @dwalsh do you concur?

Comment 5 Tom Sweeney 2021-09-07 20:00:01 UTC

and just noticed this was an 8.4 issue.  We'll probably need to backport to v3.2 and/or v3.2.3-rhel too.  Matt/Jindrich thoughts on that?

Comment 26 Joy Pu 2021-09-22 12:01:32 UTC

Test with podman-3.3.1-9.module+el8.5.0+12697+018f24d7.x86_64 and it works as expected. So set the status to verified. More details:
Image can be updated with auth files:
service file:
# container-test-auto.service
# autogenerated by Podman 3.3.1
# Wed Sep 22 07:51:06 EDT 2021

[Unit]
Description=Podman container-test-auto.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/%n.ctr-id
ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --sdnotify=conmon --cgroups=no-conmon --rm -d --replace --authfile=/root/auth.json --label io.containers.autoupdate=image --label io.containers.autoupdate.authfile=/root/auth.json --name test-auto localhost:5000/testauto top
ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target default.target

Update results:
# podman auto-update
Trying to pull localhost:5000/testauto:latest...
Getting image source signatures
Copying blob f3ac2f942260 [--------------------------------------] 0.0b / 0.0b
Copying config 9617696764 done  
Writing manifest to image destination
Storing signatures
UNIT                         CONTAINER                 IMAGE                    POLICY      UPDATED
container-test-auto.service  abb63b65d73e (test-auto)  localhost:5000/testauto  registry    true

Comment 28 errata-xmlrpc 2021-11-09 17:40:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4154