Bug 1587869

Summary: Can't import image signature from the RedHat registry
Product: OpenShift Container Platform Reporter: zhou ying <yinzhou>
Component: ImageStreamsAssignee: Ben Parees <bparees>
Status: CLOSED ERRATA QA Contact: Dongbo Yan <dyan>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.10.0CC: aos-bugs, jokerman, mmccomas, xtian, yinzhou
Target Milestone: ---   
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-30 19:17:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
controllers pod logs
none
new logs from controller none

Description zhou ying 2018-06-06 08:26:12 UTC 
Description of problem:
After config the sigstore, can't import image signatrue from the redhat registry.

Version-Release number of selected component (if applicable):
openshift v3.10.0-0.60.0

How reproducible:
always

Steps to Reproduce:
1. Config the sigstore for the master host, and restart api and controllers;
2. Try to import image from redhat registry:
`oc import-image jboss-webserver31-tomcat8-openshift:latest --from=registry.access.redhat.com/jboss-webserver-3/webserver31-tomcat8-openshift:latest --confirm -n install-test`
3. Check the image
`oc describe istag jboss-webserver31-tomcat8-openshift:latest`


Actual results:
3.Can't import the image signature from the registry.


Expected results:
3. Should import the image signature from the registry.

Additional info:
Comment 1 Ben Parees 2018-06-06 14:08:09 UTC 
can you share the contents of your /etc/containers/registries.d directory?  (filenames + contents)

w/ a redhat.yaml file in /etc/containers/registries.d with the following content:

docker:
  registry.access.redhat.com:
    sigstore: https://access.redhat.com/webassets/docker/content/sigstore


I was able to import the signatures:

Image Name:		sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e
Docker Image:		registry.access.redhat.com/jboss-webserver-3/webserver31-tomcat8-openshift@sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e
Name:			sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e
Created:		30 seconds ago
Annotations:		image.openshift.io/dockerLayersOrder=ascending
Image Size:		204.8MB (first layer 74.92MB, last binary layer 44.87MB)
Image Signatures:	 
			Name:	sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e@7955e42da12ab29746d0b18a6e92fd1f4fe68f39842dc377a86623e677149016
			Type:	AtomicImageV1
			Status:	Unverified
Image Signatures:	 
			Name:	sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e@aa244d390f5611242b06e3dc25a3e4e9fd3f59ff9a0dfb64f3cf7d3ed587f4de
			Type:	AtomicImageV1
			Status:	Unverified
Image Signatures:	 
			Name:	sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e@47763d5c2b59a8ced4548690aeb7ca252ebdfb043be9fc479436b77cef17f99b
			Type:	AtomicImageV1
			Status:	Unverified


If you can share level 5 master logs that would also help.
Comment 3 zhou ying 2018-06-07 02:54:24 UTC 
Created attachment 1448570 [details]
controllers pod logs
Comment 4 zhou ying 2018-06-07 02:55:21 UTC 
[root@qe-yinzhou-310-master-etcd-1 registries.d]# ll /etc/containers/registries.d/redhat.yaml 
-rw-r--r--. 1 root root 112 Jun  6 22:21 /etc/containers/registries.d/redhat.yaml
[root@qe-yinzhou-310-master-etcd-1 registries.d]# cat /etc/containers/registries.d/redhat.yaml 
docker:
  registry.access.redhat.com:
    sigstore: https://access.redhat.com/webassets/docker/content/sigstore
Comment 5 Ben Parees 2018-06-07 15:10:27 UTC 
No other files in /etc/containers/registries.d ?
Comment 6 zhou ying 2018-06-08 02:49:17 UTC 
With the default file /etc/containers/registries.d/default.yaml
Comment 8 zhou ying 2018-06-11 03:12:08 UTC 
Created attachment 1449843 [details]
new logs from controller
Comment 9 zhou ying 2018-06-11 03:13:50 UTC 
Ben :

   I create the env by jenkins , not sure is the same of https://bugzilla.redhat.com/show_bug.cgi?id=1506066?
Comment 10 zhou ying 2018-06-11 03:14:49 UTC 
And I can import the signature with 3.9.30.
Comment 11 Ben Parees 2018-06-11 03:18:20 UTC 
if your master is running inside a container, then the 
 /etc/containers/registries.d/redhat.yaml file needs to be present inside that container also.

So i'm not sure what the jenkins job does when creating an env, but it certainly sounds possible that the master controller process is not finding your redhat.yaml file which contains the lookaside information for the signature retrieval.

also I did not see the expected output in your logs, which makes me think that whatever openshift binary you replaced, is not the one that is running the master (which again would make sense if your master is running inside a container).
Comment 12 Ben Parees 2018-06-11 03:21:15 UTC 
oc cluster up was very different in 3.9, so if your cluster is based on oc cluster up, it may be that the configuration process has changed for this.

do you have access to a "traditional" 3.10 cluster where you can attempt signature imports?  I suspect getting the correct lookaside configuration into a master running inside a container is going to be a bit tricky, unless the new cluster up rewrite supports mounting directories into the master container.
Comment 14 Ben Parees 2018-06-11 20:39:08 UTC 
I tried to patch your cluster to mount the necessary config but i think it's not starting now.

the PR to fix this is here:
https://github.com/openshift/openshift-ansible/pull/8719

if you want to try to manually fix your cluster, you need to patch your /etc/origin/node/pods/controller.yaml w/ the hostpath mounts defined in that PR so that the /etc/containers/registries.d dir is mounted into the controller pod at the same path.
Comment 16 zhou ying 2018-06-14 05:27:43 UTC 
Confirmed with openshift ,the issue has fixed:
openshift v3.10.0-0.67.0

[root@qe-yinzhou-310-master-etcd-1 home]# oc describe istag jboss-webserver31-tomcat8-openshift:latest -n install-test
Image Name:		sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e
Docker Image:		registry.access.redhat.com/jboss-webserver-3/webserver31-tomcat8-openshift@sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e
Name:			sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e
Created:		14 minutes ago
Annotations:		image.openshift.io/dockerLayersOrder=ascending
Image Size:		204.8MB (first layer 74.92MB, last binary layer 44.87MB)
Image Signatures:	 
			Name:	sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e@7955e42da12ab29746d0b18a6e92fd1f4fe68f39842dc377a86623e677149016
			Type:	AtomicImageV1
			Status:	Unverified
Image Signatures:	 
			Name:	sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e@aa244d390f5611242b06e3dc25a3e4e9fd3f59ff9a0dfb64f3cf7d3ed587f4de
			Type:	AtomicImageV1
			Status:	Unverified
Image Signatures:	 
			Name:	sha256:29470583ce1511dbbe8e2552b7f0278d0d74595c42a00632cd23fda1ffab361e@47763d5c2b59a8ced4548690aeb7ca252ebdfb043be9fc479436b77cef17f99b
			Type:	AtomicImageV1
			Status:	Unverified
Comment 18 errata-xmlrpc 2018-07-30 19:17:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816