Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1624667

Summary: Default images are pulled from registry.access.redhat.com/openshift3 for oc adm diagnostics DiagnosticPod/AppCreate
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: ocAssignee: Luke Meyer <lmeyer>
Status: CLOSED DUPLICATE QA Contact: Xingxing Xia <xxia>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: aos-bugs, bparees, jokerman, mmccomas
Target Milestone: ---   
Target Release: 3.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-04 15:11:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2018-09-03 02:31:10 UTC
Description of problem:
oc adm diagnostics DiagnosticPod/AppCreate are pulled images from registry.access.redhat.com/openshift3, which will fail since the image is not found.
# oc adm diagnostics DiagnosticPod
[Note] Running diagnostic: DiagnosticPod
       Description: Create a pod to run diagnostics from the application standpoint
       
WARN:  [DCli2006 from diagnostic DiagnosticPod@openshift/origin/pkg/oc/cli/admin/diagnostics/diagnostics/client/pod/run_diagnostics_pod.go:187]
       Timed out preparing diagnostic pod logs for streaming, so this diagnostic cannot run.
       It is likely that the image 'registry.access.redhat.com/openshift3/ose-deployer:v3.11.0-0.25.0' was not pulled and running yet.
       Last error: (*errors.StatusError[2]) container "pod-diagnostics" in pod "pod-diagnostic-test-8bgjp" is waiting to start: image can't be pulled

Described pod info
# oc describe pod pod-diagnostic-test-8bgjp
Events:
  Type     Reason          Age                From                                   Message
  ----     ------          ----               ----                                   -------
  Normal   Scheduled       49s                default-scheduler                      Successfully assigned kube-service-catalog/pod-diagnostic-test-8bgjp to qe-zhsun-ali2master-etcd-001
  Normal   Pulling         46s                kubelet, qe-zhsun-ali2master-etcd-001  pulling image "registry.access.redhat.com/openshift3/ose-deployer:v3.11.0-0.25.0"
  Warning  Failed          45s                kubelet, qe-zhsun-ali2master-etcd-001  Failed to pull image "registry.access.redhat.com/openshift3/ose-deployer:v3.11.0-0.25.0": rpc error: code = Unknown desc = error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\""
  Warning  Failed          45s                kubelet, qe-zhsun-ali2master-etcd-001  Error: ErrImagePull
  Normal   SandboxChanged  3s (x15 over 45s)  kubelet, qe-zhsun-ali2master-etcd-001  Pod sandbox changed, it will be killed and re-created.

# oc adm diagnostics AppCreate
[Note] Running diagnostic: AppCreate
       Description: Create an application and test that it deploys correctly.
       
Info:  21:39:25.908: Using project 'openshift-diagnostic-appcreate-bs89m' for diagnostic.

ERROR: [DCluAC011 from diagnostic AppCreate@openshift/origin/pkg/oc/cli/admin/diagnostics/diagnostics/cluster/app_create/app.go:121]
       21:41:26.158: App pod was not in running state before timeout (120 sec)
       There are many reasons why this can occur; for example:
         * The app or deployer image may not be available (check pod status)
         * Downloading an image may have timed out (consider increasing timeout)
         * The scheduler may be unable to find an appropriate node for it to run (check deployer logs)
         * The node container runtime may be malfunctioning (check node and docker/cri-o logs)
       

Described pod info
oc describe pod diagnostic-appcreate-1-wgncn -n openshift-diagnostic-appcreate-7t6kg
Events:
  Type     Reason          Age               From                                   Message
  ----     ------          ----              ----                                   -------
  Normal   Scheduled       1m                default-scheduler                      Successfully assigned openshift-diagnostic-appcreate-7t6kg/diagnostic-appcreate-1-wgncn to qe-zhsun-ali2master-etcd-002
  Normal   Pulling         1m (x2 over 1m)   kubelet, qe-zhsun-ali2master-etcd-002  pulling image "registry.access.redhat.com/openshift3/ose-deployer:v3.11.0-0.25.0"
  Warning  Failed          1m (x2 over 1m)   kubelet, qe-zhsun-ali2master-etcd-002  Failed to pull image "registry.access.redhat.com/openshift3/ose-deployer:v3.11.0-0.25.0": rpc error: code = Unknown desc = error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\""
  Warning  Failed          1m (x2 over 1m)   kubelet, qe-zhsun-ali2master-etcd-002  Error: ErrImagePull
  Normal   SandboxChanged  57s (x7 over 1m)  kubelet, qe-zhsun-ali2master-etcd-002  Pod sandbox changed, it will be killed and re-created.
  Normal   BackOff         54s (x6 over 1m)  kubelet, qe-zhsun-ali2master-etcd-002  Back-off pulling image "registry.access.redhat.com/openshift3/ose-deployer:v3.11.0-0.25.0"
  Warning  Failed          54s (x6 over 1m)  kubelet, qe-zhsun-ali2master-etcd-002  Error: ImagePullBackOff



Version-Release number of selected component (if applicable):
# openshift version
openshift v3.11.0-0.25.0


How reproducible:
Always

Steps to Reproduce:
1. oc adm diagnostics DiagnosticPod
2. oc adm diagnostics AppCreate
3.

Actual results:
Default images are pulled from registry.access.redhat.com/openshift3

Expected results:
It is better to use the imageConfig in master-config.yaml

Additional info:
Could use --images parameter as workaround for oc adm diagnostics DiagnosticPod.
for example:
# oc adm diagnostics DiagnosticPod --images='registry.reg-aws.openshift.com:443/openshift3/ose-${component}:${version}'

oc adm diagnostics AppCreate does not have --images parameter, there is not workaroud

Comment 1 Juan Vallejo 2018-09-04 15:07:30 UTC
@bparees, is this something that is also fixed by https://github.com/openshift/ose/pull/1397?

Comment 2 Ben Parees 2018-09-04 15:11:02 UTC
should be, yes.

*** This bug has been marked as a duplicate of bug 1622830 ***