Bug 2218315

Summary:	podman auto-update ignores --authfile and REGISTRY_AUTH_FILE
Product:	Red Hat Enterprise Linux 8	Reporter:	Jered Floyd <jered>
Component:	podman	Assignee:	Jindrich Novy <jnovy>
Status:	CLOSED ERRATA	QA Contact:	Yuhui Jiang <yujiang>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	8.8	CC:	ajia, bbaude, dwalsh, jligon, jnovy, lsm5, mboddu, mheon, pthomas, tsweeney, umohnani, vrothber
Target Milestone:	rc	Keywords:	Triaged
Target Release:	---	Flags:	pm-rhel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	podman-4.6.0-0.3.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-11-14 15:29:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jered Floyd 2023-06-28 18:06:04 UTC

Description of problem:

podman auto-update does not use credentials provided with --authfile or specified in the REGISTRY_AUTH_FILE.

(podman auto-update also does not present an error when no credentials are available, but this may be intended behavior?) 


Version-Release number of selected component (if applicable):

4.4.1


How reproducible:

Always


Steps to Reproduce:
1. Create a new podman container with systemd integration, label io.containers.autoupdate=registry, and from a registry that requires authentication.
2. Update registry image.
3a. Run podman auto-update --authfile /srv/auth.json where /srv/auth.json has registry login credentials.  

3b. Run podman auto-update with REGISTRY_AUTH_FILE set to /srv/auth.json.

Actual results:

Container is not updated.  podman auto-update indicates "false" for Update Status.


Expected results:

Container is updated if credentials are valid and update status is "true", and update status is "failed" if credentials are unavailable or invalid.


Additional info:

Sample output (edited) with "podman auto-update --log-level debug --authfile /srv/auth.json":

time="2023-06-28T17:58:40Z" level=info msg="podman filtering at log level debug"
time="2023-06-28T17:58:40Z" level=debug msg="Called auto-update.PersistentPreRunE(podman auto-update --log-level debug --authfile /srv/auth.json)"
time="2023-06-28T17:58:40Z" level=debug msg="Using conmon: \"/usr/bin/conmon\""
time="2023-06-28T17:58:40Z" level=debug msg="Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db"
time="2023-06-28T17:58:40Z" level=debug msg="Using graph driver overlay"
time="2023-06-28T17:58:40Z" level=debug msg="Using graph root /var/lib/containers/storage"
time="2023-06-28T17:58:40Z" level=debug msg="Using run root /run/containers/storage"
time="2023-06-28T17:58:40Z" level=debug msg="Using static dir /var/lib/containers/storage/libpod"
time="2023-06-28T17:58:40Z" level=debug msg="Using tmp dir /run/libpod"
time="2023-06-28T17:58:40Z" level=debug msg="Using volume path /var/lib/containers/storage/volumes"
time="2023-06-28T17:58:40Z" level=debug msg="Using transient store: false"
time="2023-06-28T17:58:40Z" level=debug msg="Set libpod namespace to \"\""
time="2023-06-28T17:58:40Z" level=debug msg="[graphdriver] trying provided driver \"overlay\""
time="2023-06-28T17:58:40Z" level=debug msg="Cached value indicated that overlay is supported"
time="2023-06-28T17:58:40Z" level=debug msg="Cached value indicated that overlay is supported"
time="2023-06-28T17:58:40Z" level=debug msg="Cached value indicated that metacopy is being used"
time="2023-06-28T17:58:40Z" level=debug msg="Cached value indicated that native-diff is not being used"
time="2023-06-28T17:58:40Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
"
time="2023-06-28T17:58:40Z" level=debug msg="backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true"
time="2023-06-28T17:58:40Z" level=debug msg="Initializing event backend file"
time="2023-06-28T17:58:40Z" level=debug msg="Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument"
time="2023-06-28T17:58:40Z" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument"
time="2023-06-28T17:58:40Z" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument"
time="2023-06-28T17:58:40Z" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument"
time="2023-06-28T17:58:40Z" level=debug msg="Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument"
time="2023-06-28T17:58:40Z" level=debug msg="Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument"
time="2023-06-28T17:58:40Z" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument"
time="2023-06-28T17:58:40Z" level=debug msg="Using OCI runtime \"/usr/bin/runc\""
time="2023-06-28T17:58:40Z" level=info msg="Setting parallel job count to 7"
time="2023-06-28T17:58:40Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@13487ca8dd86f89158f
cdcb2127bb9b4fbab5c93b05dd4e5ec7a8a4165474cd0\""
time="2023-06-28T17:58:40Z" level=debug msg="exporting opaque data as blob \"sha256:13487ca8dd86f89158fcdcb2127bb9b4fbab5c93b05dd4e5ec7a8a4165474cd0\""
time="2023-06-28T17:58:40Z" level=debug msg="Using registries.d directory /etc/containers/registries.d"
time="2023-06-28T17:58:40Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
time="2023-06-28T17:58:40Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\""
time="2023-06-28T17:58:40Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/001-rhel-shortnames.conf\""
time="2023-06-28T17:58:41Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf\""
[..]
time="2023-06-28T17:58:43Z" level=debug msg="Trying to access \"registry.redhat.io/rhel8/mariadb-103:latest\""
time="2023-06-28T17:58:43Z" level=debug msg="No credentials matching registry.redhat.io/rhel8/mariadb-103 found in /run/user/0/containers/auth.json"
time="2023-06-28T17:58:43Z" level=debug msg="No credentials matching registry.redhat.io/rhel8/mariadb-103 found in /root/.config/containers/auth.json"
time="2023-06-28T17:58:43Z" level=debug msg="No credentials matching registry.redhat.io/rhel8/mariadb-103 found in /root/.docker/config.json"
time="2023-06-28T17:58:43Z" level=debug msg="No credentials matching registry.redhat.io/rhel8/mariadb-103 found in /root/.dockercfg"
time="2023-06-28T17:58:43Z" level=debug msg="No credentials for registry.redhat.io/rhel8/mariadb-103 found"
time="2023-06-28T17:58:43Z" level=debug msg=" Lookaside configuration: using \"docker\" namespace registry.redhat.io"
time="2023-06-28T17:58:43Z" level=debug msg="  Using \"sigstore\" https://registry.redhat.io/containers/sigstore"
time="2023-06-28T17:58:43Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io"
time="2023-06-28T17:58:43Z" level=debug msg=" Sigstore attachments: using \"docker\" namespace registry.redhat.io"
time="2023-06-28T17:58:43Z" level=debug msg="GET https://registry.redhat.io/v2/"
time="2023-06-28T17:58:43Z" level=debug msg="Ping https://registry.redhat.io/v2/ status 401"
time="2023-06-28T17:58:43Z" level=debug msg="GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?scope=repository%3Arhel8%2Fmariadb-103%3Apull&service=docker-registry"
time="2023-06-28T17:58:43Z" level=debug msg="Accessing \"registry.redhat.io/rhel8/mariadb-103:latest\" failed: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication"
time="2023-06-28T17:58:43Z" level=debug msg="Error checking updates for image registry.redhat.io/rhel8/mariadb-103: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication (ignoring error)"
[..]


Sample output (edited) with "podman auto-update --log-level debug" with the contents of /srv/auth.json in ~/.config/containers/auth.json:

time="2023-06-28T18:03:23Z" level=info msg="podman filtering at log level debug"
time="2023-06-28T18:03:23Z" level=debug msg="Called auto-update.PersistentPreRunE(podman auto-update --log-level debug)"
time="2023-06-28T18:03:23Z" level=debug msg="Using conmon: \"/usr/bin/conmon\""
time="2023-06-28T18:03:23Z" level=debug msg="Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db"
time="2023-06-28T18:03:23Z" level=debug msg="Using graph driver overlay"
time="2023-06-28T18:03:23Z" level=debug msg="Using graph root /var/lib/containers/storage"
time="2023-06-28T18:03:23Z" level=debug msg="Using run root /run/containers/storage"
time="2023-06-28T18:03:23Z" level=debug msg="Using static dir /var/lib/containers/storage/libpod"
time="2023-06-28T18:03:23Z" level=debug msg="Using tmp dir /run/libpod"
time="2023-06-28T18:03:23Z" level=debug msg="Using volume path /var/lib/containers/storage/volumes"
time="2023-06-28T18:03:23Z" level=debug msg="Using transient store: false"
time="2023-06-28T18:03:23Z" level=debug msg="Set libpod namespace to \"\""
time="2023-06-28T18:03:23Z" level=debug msg="[graphdriver] trying provided driver \"overlay\""
time="2023-06-28T18:03:23Z" level=debug msg="Cached value indicated that overlay is supported"
time="2023-06-28T18:03:23Z" level=debug msg="Cached value indicated that overlay is supported"
time="2023-06-28T18:03:23Z" level=debug msg="Cached value indicated that metacopy is being used"
time="2023-06-28T18:03:23Z" level=debug msg="Cached value indicated that native-diff is not being used"
time="2023-06-28T18:03:23Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
time="2023-06-28T18:03:23Z" level=debug msg="backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true"
time="2023-06-28T18:03:23Z" level=debug msg="Initializing event backend file"
time="2023-06-28T18:03:23Z" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument"
time="2023-06-28T18:03:23Z" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument"
time="2023-06-28T18:03:23Z" level=debug msg="Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument"
time="2023-06-28T18:03:23Z" level=debug msg="Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument"
time="2023-06-28T18:03:23Z" level=debug msg="Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument"
time="2023-06-28T18:03:23Z" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument"
time="2023-06-28T18:03:23Z" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument"
time="2023-06-28T18:03:23Z" level=debug msg="Using OCI runtime \"/usr/bin/runc\""
time="2023-06-28T18:03:23Z" level=info msg="Setting parallel job count to 7"
time="2023-06-28T18:03:23Z" level=debug msg="Looking up image \"localhost/mediawiki\" in local containers storage"
time="2023-06-28T18:03:23Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2023-06-28T18:03:23Z" level=debug msg="Trying \"localhost/mediawiki:latest\" ..."
time="2023-06-28T18:03:23Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@9fc98a366b2a73a1550bb6f1f0b4f1c6ba48305c06769b714afa475f70269ca2\""
time="2023-06-28T18:03:23Z" level=debug msg="Found image \"localhost/mediawiki\" as \"localhost/mediawiki:latest\" in local containers storage"
time="2023-06-28T18:03:23Z" level=debug msg="Found image \"localhost/mediawiki\" as \"localhost/mediawiki:latest\" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@9fc98a366b2a73a1550bb6f1f0b4f1c6ba48305c06769b714afa475f70269ca2)"
time="2023-06-28T18:03:23Z" level=debug msg="exporting opaque data as blob \"sha256:9fc98a366b2a73a1550bb6f1f0b4f1c6ba48305c06769b714afa475f70269ca2\""
time="2023-06-28T18:03:23Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@13487ca8dd86f89158fcdcb2127bb9b4fbab5c93b05dd4e5ec7a8a4165474cd0\""
time="2023-06-28T18:03:23Z" level=debug msg="exporting opaque data as blob \"sha256:13487ca8dd86f89158fcdcb2127bb9b4fbab5c93b05dd4e5ec7a8a4165474cd0\""
time="2023-06-28T18:03:23Z" level=debug msg="Using registries.d directory /etc/containers/registries.d"
time="2023-06-28T18:03:23Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
time="2023-06-28T18:03:23Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\""
time="2023-06-28T18:03:23Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/001-rhel-shortnames.conf\""
time="2023-06-28T18:03:24Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf\""
[..]
time="2023-06-28T18:03:26Z" level=debug msg="Trying to access \"registry.redhat.io/rhel8/mariadb-103:latest\""
time="2023-06-28T18:03:26Z" level=debug msg="No credentials matching registry.redhat.io/rhel8/mariadb-103 found in /run/user/0/containers/auth.json"
time="2023-06-28T18:03:26Z" level=debug msg="Found credentials for registry.redhat.io/rhel8/mariadb-103 in credential helper containers-auth.json in file /root/.config/containers/auth.json"
time="2023-06-28T18:03:26Z" level=debug msg=" Lookaside configuration: using \"docker\" namespace registry.redhat.io"
time="2023-06-28T18:03:26Z" level=debug msg="  Using \"sigstore\" https://registry.redhat.io/containers/sigstore"
time="2023-06-28T18:03:26Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io"
time="2023-06-28T18:03:26Z" level=debug msg=" Sigstore attachments: using \"docker\" namespace registry.redhat.io"
time="2023-06-28T18:03:26Z" level=debug msg="GET https://registry.redhat.io/v2/"
time="2023-06-28T18:03:26Z" level=debug msg="Ping https://registry.redhat.io/v2/ status 401"
time="2023-06-28T18:03:26Z" level=debug msg="GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=[REDACTED]&scope=repository%3Arhel8%2Fmariadb-103%3Apull&service=docker-registry"
time="2023-06-28T18:03:26Z" level=debug msg="GET https://registry.redhat.io/v2/rhel8/mariadb-103/manifests/latest"
time="2023-06-28T18:03:26Z" level=debug msg="Content-Type from manifest GET is \"application/vnd.docker.distribution.manifest.list.v2+json\""
time="2023-06-28T18:03:26Z" level=debug msg="GET https://registry.redhat.io/v2/rhel8/mariadb-103/manifests/sha256:d7ef9a47131eeea7ece6b531eb0793fe93a37fde75040defa4d777444b552b88"
time="2023-06-28T18:03:26Z" level=debug msg="Content-Type from manifest GET is \"application/vnd.docker.distribution.manifest.v2+json\""
time="2023-06-28T18:03:26Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@6701ff105db000bd2d397ff6a06ff2ced4fec2b6ae2abec736479179136ddfe5\""
time="2023-06-28T18:03:26Z" level=debug msg="exporting opaque data as blob \"sha256:6701ff105db000bd2d397ff6a06ff2ced4fec2b6ae2abec736479179136ddfe5\""
[..]

Comment 1 Valentin Rothberg 2023-06-29 06:41:41 UTC

Thanks for filing the issue, Jered!  I will take a look at it in the coming days.

Comment 2 Valentin Rothberg 2023-07-03 09:27:32 UTC

Opened the following PR upstream: https://github.com/containers/podman/pull/19092

Comment 10 errata-xmlrpc 2023-11-14 15:29:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6939