1676438 – Can't import jenkins imagestream from payload automaticly

Bug 1676438 - Can't import jenkins imagestream from payload automaticly

Summary: Can't import jenkins imagestream from payload automaticly

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Image
Sub Component:
Version:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	4.1.0
Assignee:	Gabe Montero
QA Contact:	XiuJuan Wang
Docs Contact:
URL:
Whiteboard:	beta2blocker
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-02-12 10:01 UTC by XiuJuan Wang
Modified:	2019-06-04 10:44 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:	undefined
Clone Of:
Environment:
Last Closed:	2019-06-04 10:44:00 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2019:0758	None	None	None	2019-06-04 10:44:07 UTC

Description XiuJuan Wang 2019-02-12 10:01:13 UTC

Description of problem:
Can't import jenkins imagestream from payload automaticly due to 
'secrets "coreos-pull-secret" is forbidden: User "system:serviceaccount:openshift-cluster-samples-operator:cluster-samples-operator" cannot get resource "secrets" in API group "" in the namespace "kube-system"'

Version-Release number of selected component (if applicable):

 $oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-02-12-005016   True        False         24m     Cluster version is 4.0.0-0.nightly-2019-02-12-005016
$oc get clusterversion  -o yaml   | grep image
      image: registry.svc.ci.openshift.org/ocp/release@sha256:02b57648ca23f4b9fd81567c20ad6d29e0a3824a1a0f48b4e255defa75530681

How reproducible:
always

Steps to Reproduce:
1.Check jenkins imagestream after install cluster
2.
3.

Actual results:
$ oc describe is jenkins -n openshift
Name:                   jenkins
Namespace:              openshift
Created:                20 minutes ago
Labels:                 samples.operator.openshift.io/managed=true
Annotations:            openshift.io/display-name=Jenkins
                        openshift.io/image.dockerRepositoryCheck=2019-02-12T09:13:22Z
                        samples.operator.openshift.io/version=v4.0.0-0.170.0.0-1eee5b474
Image Repository:       image-registry.openshift-image-registry.svc:5000/openshift/jenkins
Image Lookup:           local=false
Unique Images:          0
Tags:                   3
 
1
  tagged from registry.redhat.io/openshift3/jenkins-1-rhel7:latest
    prefer registry pullthrough when referencing this tag
 
  Provides a Jenkins 1.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.
  Tags: hidden, jenkins
 
  ! error: Import failed (InternalError): Internal error occurred: Get https://registry.redhat.io/v2/openshift3/jenkins-1-rhel7/manifests/latest: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
      20 minutes ago
 
2 (latest)
  tagged from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:28fca9d98f14fcd7d3444ca89388c494d3f2f407792a9336caae2717d375d885
    prefer registry pullthrough when referencing this tag
 
  Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.
  Tags: jenkins
 
  ! error: Import failed (Unauthorized): you may not have access to the Docker image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:28fca9d98f14fcd7d3444ca89388c494d3f2f407792a9336caae2717d375d885"
      20 minutes ago
 
$oc get secret -n openshift
NAME                       TYPE                                  DATA   AGE
builder-dockercfg-grgnf    kubernetes.io/dockercfg               1      30m
builder-token-876zh        kubernetes.io/service-account-token   3      30m
builder-token-zgjgh        kubernetes.io/service-account-token   3      30m
default-dockercfg-rqpx7    kubernetes.io/dockercfg               1      30m
default-token-96nzl        kubernetes.io/service-account-token   3      32m
default-token-vn8vd        kubernetes.io/service-account-token   3      30m
deployer-dockercfg-pjlst   kubernetes.io/dockercfg               1      30m
deployer-token-g8tqz       kubernetes.io/service-account-token   3      30m
deployer-token-rvgq5       kubernetes.io/service-account-token   3      30m

$ oc get config.samples.operator.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: samples.operator.openshift.io/v1
  kind: Config
  metadata:
    creationTimestamp: 2019-02-12T09:03:05Z
    finalizers:
    - samples.operator.openshift.io/finalizer
    generation: 1
    name: instance
    resourceVersion: "20122"
    selfLink: /apis/samples.operator.openshift.io/v1/configs/instance
    uid: 0275876b-2ea5-11e9-b70e-0281e682cb8a
  spec:
    architectures:
    - x86_64
    installType: rhel
    managementState: Managed
    samplesRegistry: registry.access.redhat.com
  status:
    architectures:
    - x86_64
    conditions:
    - lastTransitionTime: 2019-02-12T09:05:10Z
      lastUpdateTime: 2019-02-12T09:05:10Z
      status: "True"
      type: ConfigurationValid
    - lastTransitionTime: 2019-02-12T09:13:53Z
      lastUpdateTime: 2019-02-12T09:13:53Z
      message: '<imagestream/jenkins-agent-nodejs>you may not have access to the Docker
        image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66d0c3f08119b82f320abb2f589f2a786c394ba7a85ca0b8efcd7818d49a8be5"<imagestream/jenkins-agent-nodejs><imagestream/jenkins-agent-maven>you
        may not have access to the Docker image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9feaf46a241502b5cd52009cc23b5370891d549826a84a0a8608e64d4952137"<imagestream/jenkins-agent-maven><imagestream/jenkins>Internal
        error occurred: Get https://registry.redhat.io/v2/openshift3/jenkins-1-rhel7/manifests/latest:
        unauthorized: Please login to the Red Hat Registry using your Customer Portal
        credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531<imagestream/jenkins>'
      reason: 'jenkins-agent-nodejs jenkins-agent-maven jenkins '
      status: "True"
      type: ImportImageErrorsExist
    - lastTransitionTime: 2019-02-12T09:05:19Z
      lastUpdateTime: 2019-02-12T09:05:19Z
      status: "True"
      type: SamplesExist
    - lastTransitionTime: 2019-02-12T09:13:59Z
      lastUpdateTime: 2019-02-12T09:13:59Z
      message: 'error creating samples: an error on the server ("Internal Server Error:
        \"/apis/template.openshift.io/v1/namespaces/openshift/templates\": Post https://172.30.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews:
        dial tcp 172.30.0.1:443: connect: connection refused") has prevented the request
        from succeeding (post templates.template.openshift.io)'
      status: "False"
      type: ImageChangesInProgress
    - lastTransitionTime: 2019-02-12T09:14:02Z
      lastUpdateTime: 2019-02-12T09:14:02Z
      message: 'secrets "coreos-pull-secret" is forbidden: User "system:serviceaccount:openshift-cluster-samples-operator:cluster-samples-operator"
        cannot get resource "secrets" in API group "" in the namespace "kube-system"'
      status: "False"
      type: ImportCredentialsExist
    - lastTransitionTime: 2019-02-12T09:05:10Z
      lastUpdateTime: 2019-02-12T09:05:10Z
      status: "False"
      type: RemovePending
    - lastTransitionTime: 2019-02-12T09:05:10Z
      lastUpdateTime: 2019-02-12T09:05:10Z
      status: "False"
      type: MigrationInProgress
    installType: rhel
    managementState: Managed
    samplesRegistry: registry.access.redhat.com
    version: v4.0.0-0.170.0.0-1eee5b474
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Expected results:


Additional info:

$oc get sa -n openshift-cluster-samples-operator  cluster-samples-operator -o yaml 
apiVersion: v1
imagePullSecrets:
- name: cluster-samples-operator-dockercfg-2kxdp
kind: ServiceAccount
metadata:
  creationTimestamp: 2019-02-12T09:02:28Z
  name: cluster-samples-operator
  namespace: openshift-cluster-samples-operator
  resourceVersion: "11616"
  selfLink: /api/v1/namespaces/openshift-cluster-samples-operator/serviceaccounts/cluster-samples-operator
  uid: ec8e6af8-2ea4-11e9-b93b-0a027c84da50
secrets:
- name: cluster-samples-operator-token-l6mnh
- name: cluster-samples-operator-dockercfg-2kxdp

Comment 1 Ben Parees 2019-02-12 15:01:15 UTC

Can you run "oc import-image jenkins --all --confirm -n openshift" and see if it imports successfully?  i'd like to understand if the credentials are invalid, or if we just had a timing issue.

Comment 2 Ben Parees 2019-02-12 15:10:21 UTC

ah, i missed this at the top:

Can't import jenkins imagestream from payload automaticly due to 
'secrets "coreos-pull-secret" is forbidden: User "system:serviceaccount:openshift-cluster-samples-operator:cluster-samples-operator" cannot get resource "secrets" in API group "" in the namespace "kube-system"'


that would imply the operator failed to copy the creds correctly, so that would cause the import failure.  We'll have to double check the RBAC roles.

Comment 3 Gabe Montero 2019-02-12 15:11:15 UTC

Also, if you are running a level with https://github.com/openshift/cluster-samples-operator/pull/98 you should 
see credentials the samples registry credentials in the openshift namespace (we copy the coreos pull secret 
by default with that change ... you don't have to manually create the credential in the operator namespace

If you see that secret, then yeah, the pull secret is invalid as Ben speculated

If you do not see that secret, we'll want you to retry once https://github.com/openshift/cluster-samples-operator/pull/98
has made it into a cluster you can try.

Comment 4 Gabe Montero 2019-02-12 15:12:55 UTC

ah missed that too

we could double check, but we are also changing the RBAC with https://bugzilla.redhat.com/show_bug.cgi?id=1675135

closing this as dup of that might make sense then

Comment 5 Gabe Montero 2019-02-12 15:23:40 UTC

Ben is investigating how we can get an OCP based installed, but in the interim, XiuJuan, can you run:

oc get roles cluster-samples-operator-kube-system-edit -n kube-system -o yaml

and report back what you get?

Comment 6 Gabe Montero 2019-02-12 15:25:09 UTC

sorry, that is 

oc get rolebinding cluster-samples-operator-kube-system-edit -n kube-system -o yaml

Comment 7 Gabe Montero 2019-02-12 16:10:12 UTC

I was able to duplicate with the origin install .... we figured out the manifests need to be re-ordered

Comment 8 Gabe Montero 2019-02-12 16:15:19 UTC

PR https://github.com/openshift/cluster-samples-operator/pull/101 is up

Comment 9 Gabe Montero 2019-02-12 17:30:39 UTC

PR has merged

Comment 11 XiuJuan Wang 2019-02-14 03:26:56 UTC

Jenkins imagestream could be imported automaticly.
$ oc describe is jenkins  -n openshift
Name:			jenkins
Namespace:		openshift
Created:		45 minutes ago
Labels:			samples.operator.openshift.io/managed=true
Annotations:		openshift.io/display-name=Jenkins
			samples.operator.openshift.io/version=v4.0.0-0.171.0.1-f15a89623
Image Repository:	image-registry.openshift-image-registry.svc:5000/openshift/jenkins
Image Lookup:		local=false
Unique Images:		1
Tags:			2

2 (latest)
  tagged from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:946d88f4c19ce9952f3fc44fcab7fdd15015dc91a57f7788fdfb0546046db90c
    prefer registry pullthrough when referencing this tag

  Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.
  Tags: jenkins

  * quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:946d88f4c19ce9952f3fc44fcab7fdd15015dc91a57f7788fdfb0546046db90c
      45 minutes ago
$ oc get secret  -n openshift
NAME                           TYPE                                  DATA   AGE
builder-dockercfg-qmwqs        kubernetes.io/dockercfg               1      46m
builder-token-h75lf            kubernetes.io/service-account-token   3      46m
builder-token-l29qq            kubernetes.io/service-account-token   3      46m
default-dockercfg-kgdjc        kubernetes.io/dockercfg               1      46m
default-token-4fkgf            kubernetes.io/service-account-token   3      49m
default-token-z5hlq            kubernetes.io/service-account-token   3      46m
deployer-dockercfg-2mdjc       kubernetes.io/dockercfg               1      46m
deployer-token-vlfg6           kubernetes.io/service-account-token   3      46m
deployer-token-xjk8s           kubernetes.io/service-account-token   3      46m
samples-registry-credentials   kubernetes.io/dockerconfigjson        1      46m

$oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-02-13-204401   True        False         43m     Cluster version is 4.0.0-0.nightly-2019-02-13-204401

Comment 14 errata-xmlrpc 2019-06-04 10:44:00 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758

Note You need to log in before you can comment on or make changes to this bug.