Bug 1613727 – need to get image_registry from env

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1613727 - need to get image_registry from env

Summary:	need to get image_registry from env

Status:	CLOSED ERRATA

Aliases:	None

Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Monitoring (Show other bugs)
Sub Component:
Version:	3.11.0
Hardware:	Unspecified Unspecified

Priority	unspecified Severity medium
Target Milestone:	---
Target Release:	3.11.0
Assigned To:	Frederic Branczyk
QA Contact:	Junqi Zhao
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2018-08-08 04:48 EDT by Junqi Zhao
Modified:	2018-10-11 03:24 EDT (History)
CC List:	1 user (show)

See Also:
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2018-10-11 03:24:08 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2018:2652	None	None	None	2018-10-11 03:24 EDT

Groups: None (edit)

Description Junqi Zhao 2018-08-08 04:48:06 EDT

Description of problem:

In roles/openshift_facts/defaults/main.yml
system_images_registry_dict:
  openshift-enterprise: "registry.redhat.io"
  origin: "docker.io"
system_images_registry: "{{ system_images_registry_dict[openshift_deployment_type | default('origin')] }}"

and https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_cluster_monitoring_operator/defaults/main.yml#L4
l_openshift_cluster_monitoring_operator_ocp_image_registry: "{{ system_images_registry_dict[openshift_deployment_type] }}/openshift3/"

if openshift_deployment_type is openshift-enterprise, the image prefix would be registry.redhat.io/openshift3/

since only the released images would push to registry.redhat.io, for not released images, there will be error for cluster-monitoring-operator pod: Back-off pulling image "registry.redhat.io/openshift3/ose-cluster-monitoring-operator:v3.11"

    kubectl -n openshift-monitoring get pods
    NAME                                          READY     STATUS             RESTARTS   AGE
    cluster-monitoring-operator-8c5ffc74f-nvxmz   0/1       ImagePullBackOff   0          1h

    kubectl -n openshift-monitoring describe pod cluster-monitoring-operator-8c5ffc74f-nvxmz
    Events:
      Type     Reason          Age                From                               Message
      ----     ------          ----               ----                               -------
      Normal   Scheduled       1h                 default-scheduler                  Successfully assigned openshift-monitoring/cluster-monitoring-operator-8c5ffc74f-nvxmz to qe-juzhao-311-qeos-nrr-1
      Normal   SandboxChanged  1h                 kubelet, qe-juzhao-311-qeos-nrr-1  Pod sandbox changed, it will be killed and re-created.
      Normal   Pulling         1h (x4 over 1h)    kubelet, qe-juzhao-311-qeos-nrr-1  pulling image "registry.redhat.io/openshift3/ose-cluster-monitoring-operator:v3.11"
      Warning  Failed          1h (x4 over 1h)    kubelet, qe-juzhao-311-qeos-nrr-1  Failed to pull image "registry.redhat.io/openshift3/ose-cluster-monitoring-operator:v3.11": rpc error: code = Unknown desc = Get https://registry.redhat.io/v2/openshift3/ose-cluster-monitoring-operator/manifests/v3.11: unauthorized: An internal server error occurred that prevented authentication operations from succeeding.
      Warning  Failed          1h (x4 over 1h)    kubelet, qe-juzhao-311-qeos-nrr-1  Error: ErrImagePull
      Normal   BackOff         7m (x386 over 1h)  kubelet, qe-juzhao-311-qeos-nrr-1  Back-off pulling image "registry.redhat.io/openshift3/ose-cluster-monitoring-operator:v3.11"
      Warning  Failed          2m (x407 over 1h)  kubelet, qe-juzhao-311-qeos-nrr-1  Error: ImagePullBackOff

So we need to get image prefix from env, you can refer to this file
roles/openshift_web_console/defaults/main.yml
openshift_web_console_image: "{{ l_osm_registry_url | regex_replace('${component}' | regex_escape, 'web-console') }}"


Version-Release number of selected component (if applicable):
# rpm -qa | grep ansible
openshift-ansible-roles-3.11.0-0.11.0.git.0.3c66516None.noarch
ansible-2.6.2-1.el7ae.noarch
openshift-ansible-docs-3.11.0-0.11.0.git.0.3c66516None.noarch
openshift-ansible-playbooks-3.11.0-0.11.0.git.0.3c66516None.noarch
openshift-ansible-3.11.0-0.11.0.git.0.3c66516None.noarch


How reproducible:
Always

Steps to Reproduce:
1. Install cluster monitoring
2.
3.

Actual results:
Pulled images from registry.redhat.io/openshift3

Expected results:
need to get image_registry from env

Additional info:

Comment 1 Frederic Branczyk 2018-08-08 05:57:17 EDT

I just opened https://github.com/openshift/openshift-ansible/pull/9477. Could you please test head of that pull request to see if it's fixed? Thanks!

Comment 2 Junqi Zhao 2018-08-09 02:33:06 EDT

(In reply to Frederic Branczyk from comment #1)
> I just opened https://github.com/openshift/openshift-ansible/pull/9477.
> Could you please test head of that pull request to see if it's fixed? Thanks!

issue if fixed, image registry could be got from env, but due to Bug 1613689, prometheus-operator pod could not start up with OCP images.

PR could be merged now, thanks

Comment 3 Frederic Branczyk 2018-08-13 13:36:09 EDT

https://github.com/openshift/openshift-ansible/pull/9477 is now merged, so this should be fixed, but I also opened https://github.com/openshift/openshift-ansible/pull/9567 to fix the issues raised in https://bugzilla.redhat.com/show_bug.cgi?id=1613689. If you could test the head of that new pull request, that would be great. Thanks!

Comment 4 Junqi Zhao 2018-08-14 02:43:36 EDT

(In reply to Frederic Branczyk from comment #3)
> https://github.com/openshift/openshift-ansible/pull/9477 is now merged, so
> this should be fixed, but I also opened
> https://github.com/openshift/openshift-ansible/pull/9567 to fix the issues
> raised in https://bugzilla.redhat.com/show_bug.cgi?id=1613689. If you could
> test the head of that new pull request, that would be great. Thanks!

I only can verify it uses ose-configmap-reloader image now
- --config-reloader-image=registry.dev.redhat.io/openshift3/ose-configmap-reloader:v3.11.0

But prometheus-operator-9f6cffdb-vrrtf pod is in CrashLoopBackOff status,
will open another bug to track
# kubectl -n openshift-monitoring get pod
NAME                                          READY     STATUS             RESTARTS   AGE
cluster-monitoring-operator-9f7578d96-c2m8p   1/1       Running            0          49m
prometheus-operator-9f6cffdb-vrrtf            0/1       CrashLoopBackOff   13         47m

Comment 5 Junqi Zhao 2018-08-14 03:01:35 EDT

Bug in Comment 4, please see Bug 1615732

Comment 6 Junqi Zhao 2018-08-16 00:08:05 EDT

@Frederic
# kubectl -n openshift-monitoring get pod | grep grafana
grafana-7476cc5c4b-pkxpb                       2/2       Running   0          1h

grafana image name is grafana, but according to [1], "Required card information" part, it seems the image name should be ose-grafana. I am not sure what the image name should be
# kubectl -n openshift-monitoring get deployment.apps/grafana -o yaml  | grep -i image
        image: registry.reg-aws.openshift.com:443/openshift3/grafana:v3.11
        imagePullPolicy: IfNotPresent
        image: registry.reg-aws.openshift.com:443/openshift3/oauth-proxy:v3.11
        imagePullPolicy: IfNotPresent
*********************************************************************************

[1] https://trello.com/c/zBEHmlss/647-buildauto-build-openshift-monitoring-images

Comment 7 Frederic Branczyk 2018-08-16 04:08:56 EDT

Yes sorry, that was misinformation. I was under the impression that Grafana was not previously built for OCP, but it was and it did not have the `ose-` prefix, so we decided to continue to use that name. Meaning this is all correct and as intended, I apologize for the confusion.

Comment 8 Junqi Zhao 2018-08-22 22:02:51 EDT

Could be changed to ON_QA, issue is fixed

Comment 9 Junqi Zhao 2018-08-22 22:08:45 EDT

image_registry is got from env now

openshift-ansible version:
openshift-ansible-3.11.0-0.20.0.git.0.ec6d8caNone.noarch.rpm
openshift-ansible-docs-3.11.0-0.20.0.git.0.ec6d8caNone.noarch.rpm
openshift-ansible-playbooks-3.11.0-0.20.0.git.0.ec6d8caNone.noarch.rpm
openshift-ansible-roles-3.11.0-0.20.0.git.0.ec6d8caNone.noarch.rpm

Comment 11 Junqi Zhao 2018-08-23 03:23:36 EDT

Per Comment 8 - Comment 9, set it to VERIFIED

Comment 13 errata-xmlrpc 2018-10-11 03:24:08 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652

Note You need to log in before you can comment on or make changes to this bug.