Bug 2024859 - Not able to deploy an existing image from private image registry using developer console
Summary: Not able to deploy an existing image from private image registry using develo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.9
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Oleg Bulatov
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On:
Blocks: 2042677
TreeView+ depends on / blocked
 
Reported: 2021-11-19 10:04 UTC by JS
Modified: 2023-01-16 08:02 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the registry internally resolved docker.io references into registry-1.docker.io and used it to find credentials Consequence: it couldn't find credentials for docker.io images Fix: normalize the registry-1.docker.io host name back to docker.io before looking up for credentials. Result: the registry can correctly find credentials for docker.io images
Clone Of:
Environment:
Last Closed: 2022-03-10 16:29:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
quick screenshot (222.73 KB, image/png)
2021-11-19 10:04 UTC, JS
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift image-registry pull 299 0 None Merged Bug 2024859: Fix auth for docker.io images 2022-01-05 22:58:06 UTC
Red Hat Knowledge Base (Solution) 6528551 0 None None None 2022-01-31 12:26:32 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:30:04 UTC

Description JS 2021-11-19 10:04:13 UTC
Created attachment 1842704 [details]
quick screenshot

Description of problem:
Using "developer" view, one is not able to successfully deploy an app using a personal account from docker.io (might affect others registries that need creds)

Version-Release number of selected component (if applicable):
OCP 4.9.5


How reproducible:
Always

Steps to Reproduce:
1. Login OCP
2. Shift to "Developer" view
3. Hit "+Add" button
4. Hit "Container Image" tile
5. Fill "Image name from external registry" with a docker.io account (tested): `docker.io/<username>/<image>`
6. Hit "create an Image pull secret" to address auth for your docker.io account
7. Enter default values for the rest

Actual results:
Check app log output and you will find errors: `errors: denied: requested access to the resource is denied unauthorized: authentication required`

Created secret it's available upon creation on previous menu

Expected results:
Smoothly deploy the app from docker.io with new created creds


Additional info:
Tested this workflow on OCP 4.7.30: success
Tested this workflow on OCP 4.9.5: error

Comment 1 JS 2021-11-19 10:41:59 UTC
Tested with OCP 4.9.7: Issue persists

Comment 3 XiuJuan Wang 2021-11-23 09:40:59 UTC
Test the pr
Follow the steps to test
1. Login OCP
2. Shift to "Developer" view
3. Hit "+Add" button
4. Hit "Container Image" tile
5. Fill "Image name from external registry" with a docker.io account (tested): `docker.io/<username>/<image>`
6. Hit "create an Image pull secret" to address auth for your docker.io account
7. Enter default values for the rest

Could pull the private image after add secret in web console
$oc get secret
jenkins-test               kubernetes.io/dockerconfigjson        1      3m6s

$oc get is 
 NAME     IMAGE REPOSITORY                                              TAGS     UPDATED
docker   image-registry.openshift-image-registry.svc:5000/wxj/docker   latest   41 seconds ago

Check pod
  Normal  Pulled          3m17s  kubelet            Successfully pulled image "image-registry.openshift-image-registry.svc:5000/wxj/docker@sha256:0d0160ac4122ab4aabdedc4ae8ff2df945895060cbb501fc96dcc48b6fb35ea4" in 13.22257265s

Comment 6 Scott Dodson 2021-12-09 03:49:33 UTC
Based on comment 0 saying this worked in 4.7 and fails in 4.9, this appears to be a regression. We're currently evaluating promoting upgrades from 4.8 to 4.9 to the stable channel and would like to understand the impact. What follows are our standard impact assessment questions. Please reply answering this questions to the best of your knowledge.

----

We're asking the following questions to evaluate whether or not this bug warrants blocking an upgrade edge from either the previous X.Y or X.Y.Z. The ultimate goal is to avoid delivering an update which introduces new risk or reduces cluster functionality in any way. Sample answers are provided to give more context and the ImpactStatementRequested label has been added to this bug. When responding, please remove ImpactStatementRequested and set the ImpactStatementProposed label. The expectation is that the assignee answers these questions.

Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking?

    example: Customers upgrading from 4.y.Z to 4.y+1.z running on GCP with thousands of namespaces, approximately 5% of the subscribed fleet
    example: All customers upgrading from 4.y.z to 4.y+1.z fail approximately 10% of the time

What is the impact? Is it serious enough to warrant blocking edges?

    example: Up to 2 minute disruption in edge routing
    example: Up to 90 seconds of API downtime
    example: etcd loses quorum and you have to restore from backup

How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)?

    example: Issue resolves itself after five minutes
    example: Admin uses oc to fix things
    example: Admin must SSH to hosts, restore from backups, or other non standard admin activities

Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)?

    example: No, it has always been like this we just never noticed
    example: Yes, from 4.y.z to 4.y+1.z Or 4.y.z to 4.y.z+1

Comment 7 W. Trevor King 2021-12-09 05:27:49 UTC
Adding ImpactStatementRequested, per [1].  When proposing an impact statement, remove ImpactStatementRequested and set ImpactStatementProposed.

[1]: https://github.com/openshift/enhancements/tree/master/enhancements/update/update-blocker-lifecycle#proposal

Comment 8 Oleg Bulatov 2021-12-10 17:17:22 UTC
Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking?

Customers who use 4.8.18+, 4.9.0+

What is the impact? Is it serious enough to warrant blocking edges?

The registry cannot find credentials for docker.io in the secrets, pull-through from private docker.io repos is broken.

How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)?

The registry expects registry-1.docker.io as the registry name for docker.io references, so a new secrets may be created.
Or the images may be mirrored into the registry (using `oc image mirror` or skopeo), so that the registry doesn't need to use pull-through.

Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)?

Yes, from 4.8.17 to 4.8.18 and in 4.9.0.

Comment 9 W. Trevor King 2022-01-19 22:02:07 UTC
Based on comment 8's impact statement, this sounds like it could be significant for some workloads.  And we're not aware of particularly many cases where folks were actually bitten by this, although we don't have Telemetry/Insights coverage for "what fraction of the fleet is being bitten?", so this is mostly word of mouth customer cases and such.  Because dropping an update recommendation impacts the whole fleet, even clusters that would not be impacted by this issue, and because there are workarounds as discussed in comment 8, we're going to drop the UpgradeBlocker keyword for now.  But if folks want to re-evaluate severity and impact here, feel free to add it back with any new evidence.

Comment 12 errata-xmlrpc 2022-03-10 16:29:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.