Bug 1749256 - Cannot prune registry 'invalid resource name "psd2-anusha/psd2-anusha": [may not contain '/']'
Summary: Cannot prune registry 'invalid resource name "psd2-anusha/psd2-anusha": [may ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.3.0
Assignee: Ricardo Maraschini
QA Contact: Wenjing Zheng
URL:
Whiteboard:
: 1752513 (view as bug list)
Depends On:
Blocks: 1752513 1755780 1766110 1766113
TreeView+ depends on / blocked
 
Reported: 2019-09-05 08:46 UTC by Sergio G.
Modified: 2020-02-21 07:15 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: older imagestreams could have an invalid name Consequence: image pruning failed because the specs for the imagestream's tags were not valid Fix: image pruner will always prune images if the associated imagestream has an invalid name Result: image pruning completes if imagestreams with invalid names are present
Clone Of:
: 1752513 1755780 1766113 (view as bug list)
Environment:
Last Closed: 2020-01-23 11:05:47 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift image-registry pull 194 'None' closed Bug 1749256: Checking if image name is valid, prune otherwise. 2020-02-21 06:39:01 UTC
Red Hat Product Errata RHBA-2020:0062 None None None 2020-01-23 11:06:11 UTC

Description Sergio G. 2019-09-05 08:46:00 UTC
Description of problem:
Cannot run prune in the registry. It fails with:
ERRO[1760] failed to get the image stream dk0502-a/psd2-anusha/psd2-anusha: invalid resource name "psd2-anusha/psd2-anusha": [may not contain '/']


Version-Release number of selected component (if applicable):
3.9


How reproducible:
Every single time


Steps to Reproduce:
1. oc exec docker-registry-NAME -- /bin/sh -c 'REGISTRY_LOG_LEVEL=info /usr/bin/dockerregistry -prune=check' 


Actual results:
ERRO[1760] failed to get the image stream dk0502-a/psd2-anusha/psd2-anusha: invalid resource name "psd2-anusha/psd2-anusha": [may not contain '/']


Expected results:
Prune finishes correctly.


Additional info:
Some troubleshooting has been done, we found missing references to a previous image which was wrong and was deleted. We also deleted the imagestream (and we will push it back again) but then the problem reproduced with another image so we must confirm what's going on under the hood before continuing deleting images.

Comment 1 Sergio G. 2019-09-05 08:49:50 UTC
Some troubleshooting:
$ oc delete is psd2-anusha
imagestream "psd2-anusha" deleted

$ oc get is
NAME               DOCKER REPO                                                  TAGS      UPDATED
psd2-karthik-002   docker-registry.default.svc:5000/dk0502-a/psd2-karthik-002   1.0.0     14 months ago
psd2-sujit         docker-registry.default.svc:5000/dk0502-a/psd2-sujit         1.0.0     13 months ago

$ oc get istag
NAME                     DOCKER REF                                                                                                              UPDATED         IMAGENAME
psd2-karthik-002:1.0.0   172.30.154.106:5000/dk0502-a/psd2-karthik-002@sha256:fad833b733493a7ae7a8cb95039b1e563a31838a015f0bd683cd6a5561fec457   14 months ago   sha256:fad833b733493a7ae7a8cb95039b1e563a31838a015f0bd683cd6a5561fec457
psd2-sujit:1.0.0         172.30.154.106:5000/dk0502-a/psd2-sujit@sha256:6184d37ebe6cd2de51feb930ac5f628aae3ba7905e6680d188c812719c000302         13 months ago   sha256:6184d37ebe6cd2de51feb930ac5f628aae3ba7905e6680d188c812719c000302

So that image stream is no more in dk0502-a project and we expected to succeed but we got the same error:
$ oc exec docker-registry-NAME -- /bin/sh -c 'REGISTRY_LOG_LEVEL=info /usr/bin/dockerregistry -prune=check'
...
INFO[1759] Would delete manifest link: dk0501-a/virtuoso@sha256:ea4eacba0302d418cb05f11215a5cdd50f51f2422910c9f869f7d39459c58476  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587
INFO[1759] Would delete manifest link: dk0501-a/virtuoso72@sha256:09c75b8cca710ab86ea46c1b128608bfee88eec80f7da3199d85540560cfd345  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587
INFO[1759] Would delete manifest link: dk0501-a/virtuoso72@sha256:1f298418e82d72e5339d3c5fb943499d72e7d2627dd7e30caa8eff3faf4c5db5  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587
INFO[1759] Would delete manifest link: dk0501-a/virtuoso72@sha256:2ae3d1a894a2722a68ae271adffefaf6b4b0fe885fcb6e55e97aaef12ea42b6b  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587
INFO[1760] Would delete manifest link: dk0501-a/virtuoso72@sha256:7400245604988798f638333d573d88eeef386a813d73e065ab08765be7fe93aa  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587
INFO[1760] Would delete manifest link: dk0501-a/virtuoso72@sha256:a59ace00e6759f2f625a1147e0e3d4b948ceb278c260cf9dde0ee933c123d063  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587
INFO[1760] Would delete manifest link: dk0501-a/virtuoso72@sha256:b069c5798a9644c3727783e96a5dcc391bcb24154f641b20f401dcbefce95b49  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587
INFO[1760] Would delete manifest link: dk0501-a/virtuoso72@sha256:c81398875cb5bfbb27174dda51c11c512f2e4a75689efe4f9a46124d4378389c  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587
INFO[1760] Would delete manifest link: dk0501-a/virtuoso72@sha256:ee587a9ab56ec595ae495b9364843d245befb73a48a6d5408c1646b62bd91e69  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587
INFO[1760] Would delete manifest link: dk0501-a/virtuoso72@sha256:f929d02383e05eba1bf40efc7dc31cb626cc43bb2648ea51d4179db314e83a1c  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587
INFO[1760] The image stream dk0502-a/psd2-anusha is not found, will remove the whole repository  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587
INFO[1760] Would delete manifest link: dk0501-a/virtuoso72@sha256:f9a32215c227c3298e3f90440798135469fd5d3e24d20c6b6ec5f1f142c94293  go.version=go1.9.4 instance.id=2b65daf4-7012-4cd6-a918-0ce94ad26587

ERRO[1760] failed to get the image stream dk0502-a/psd2-anusha/psd2-anusha: invalid resource name "psd2-anusha/psd2-anusha": [may not contain '/']
Would delete 0 blobs
Would free up 0B of disk space

Then we deleted the directory directly from the /registry filesystem and try again:
$ oc exec docker-registry-NAME -- /bin/sh -c 'REGISTRY_LOG_LEVEL=info /usr/bin/dockerregistry -prune=check'
...
ERRO[2347] failed to get the image stream dk0709-d/jupyter/minimal-notebook: invalid resource name "jupyter/minimal-notebook": [may not contain '/']

Same error with another image.

Is there any way to know what is happening and not just keep deleting images?

Comment 10 Ricardo Maraschini 2019-09-16 15:55:07 UTC
*** Bug 1752513 has been marked as a duplicate of this bug. ***

Comment 13 Wenjing Zheng 2019-10-24 07:18:10 UTC
Steps:
1. Import some image, it'll create something like /docker/registry/v2/repositories/NAMESPACE/NAME 
2. Copy these files as docker/registry/v2/repositories/foo/bar/baz
3. $ oc adm policy add-cluster-role-to-user system:image-pruner system:serviceaccount:openshift-image-registry:registry
4. $ oc exec image-registry-98888df69-j7gs7 -- /bin/sh -c 'REGISTRY_LOG_LEVEL=info /usr/bin/dockerregistry -prune=check'
time="2019-10-24T07:15:55.362046985Z" level=info msg="start prune (dry-run mode)" distribution_version=v2.6.0+unknown go.version=go1.12.8 openshift_version=v4.3.0-201910221117+c34a783-dirty
time="2019-10-24T07:15:55.749814531Z" level=info msg="Invalid image name foo/bar/ruby-ex, removing whole repository" go.version=go1.12.8
time="2019-10-24T07:15:56.0891229Z" level=info msg="Would delete repository: foo/bar/ruby-ex" go.version=go1.12.8
Would delete 0 blobs
Would free up 0B of disk space
Use -prune=delete to actually delete the data

Verified on 4.3.0-0.nightly-2019-10-23-150053

Comment 15 errata-xmlrpc 2020-01-23 11:05:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.