Bug 1581622
| Summary: | [CNS][3.10] Installation failed due to GlusterFS pods try to pull image from docker.io | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Wenkai Shi <weshi> | |
| Component: | Installer | Assignee: | Jose A. Rivera <jarrpa> | |
| Status: | CLOSED ERRATA | QA Contact: | Wenkai Shi <weshi> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 3.10.0 | CC: | aos-bugs, bkunal, bleanhar, jarrpa, jokerman, mmccomas, nberry, sdodson, sjenning, tatanaka, weshi | |
| Target Milestone: | --- | |||
| Target Release: | 3.10.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | No Doc Update | ||
| Doc Text: |
undefined
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1583148 (view as bug list) | Environment: | ||
| Last Closed: | 2018-07-30 19:16:18 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1583500 | |||
| Bug Blocks: | 1583148 | |||
We haven't changed anything about this from 0.47 to 0.50. Did something change with regard to how the default registry is configured? Wenkai, Any chance that openshift_storage_glusterfs_image was set in your previous run but not these? I agree that there haven't been any changes to this code in quite some time. (In reply to Scott Dodson from comment #4) > Wenkai, > > Any chance that openshift_storage_glusterfs_image was set in your previous > run but not these? I agree that there haven't been any changes to this code > in quite some time. Hi Scott, I didn't set openshift_storage_glusterfs_image in any of this bug's installation. AS I mentioned, it may related to atomic-openshift rather than openshift-ansible, I've try deploy atomic-openshift-3.10.0-0.47.0.git.0.2fffa04.el7 with openshift-ansible-3.10.0-0.50.0.git.0.bd68ade.el7, It's doesn't appear. This issue does appear in atomic-openshift-3.10.0-0.50.0.git.0.db6dfd6.el7 so far. Give a shoot with version atomic-openshift-3.10.0-0.51.0.git.0.8bcf033.el7, still appear. Jose, have you been able to reproduce this bug by any chance? It would be very helpful to try Wenkai's reproducer and confirm it did indeed actually start happening after 3.10.0-0.47.0. I have not been able to reproduce this, I don't have a reproducible downstream environment to test on. Moving this back to modified pending further results from Wenkai. I'll try to get a downstream environment running somewhere. It could be work around by set following parameters: openshift_storage_glusterfs_heketi_image=registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7 openshift_storage_glusterfs_image=registry.access.redhat.com/rhgs3/rhgs-server-rhel7 openshift_storage_glusterfs_block_image=registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7 openshift_storage_glusterfs_s3_image=registry.access.redhat.com/rhgs3/rhgs-gluster-s3-server-rhel7 It's related: https://github.com/openshift/openshift-ansible/pull/8585 Seth, is this releated to https://bugzilla.redhat.com/show_bug.cgi?id=1583500 ? Wenkai, This problem should no longer exist in openshift-ansible-3.10.0-0.57.0 as long as oreg_url is set, have you tested this version? Neha, Brenton, Yes, but by default the installer will set fully qualified names, the inventory you're using bypasses this. I'd suggest setting fully qualified names for these variables you're setting : openshift_storage_glusterfs_heketi_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhgs3/rhgs-volmanager-rhel7' openshift_storage_glusterfs_block_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhgs3/rhgs-gluster-block-prov-rhel7' openshift_storage_glusterfs_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhgs3/rhgs-server-rhel7' Relying on the additional registries functionality is something we should put a stop to in 3.10 at least in installer code. For origin it may be wise to preserve existing behavior while making it clear that it's deprecated. (In reply to Scott Dodson from comment #16) > Wenkai, > > This problem should no longer exist in openshift-ansible-3.10.0-0.57.0 as > long as oreg_url is set, have you tested this version? > Yes, I've tested this version. I can verify it once move to ON_QA. Moving ON_QA Verified with version openshift-ansible-3.10.0-0.64.0.git.20.48df973.el7, it's can pull docker image from registry defined in oreg_url. The customer is facing the similar issue when installing CNS with OpenShift 3.9. Could you tell me setting the following parameters can be worked around?
openshift_storage_glusterfs_heketi_image=registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7
openshift_storage_glusterfs_image=registry.access.redhat.com/rhgs3/rhgs-server-rhel7
openshift_storage_glusterfs_block_image=registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7
openshift_storage_glusterfs_s3_image=registry.access.redhat.com/rhgs3/rhgs-gluster-s3-server-rhel7
After setting these parameteres, the customer now faces the following error.
> exception: 'glusterfs_heketi_url' is undefined
Takayoshi, The problems in 3.9 are not the same as what's being discussed in this bug or at least do not have the same fix as they will on 3.10. Can you please open a new bug with as much log output as possible and the inventory file you're using when you encounter the error on 3.9? Those configuration variables should be fine. Scott, Thank you for the reply. We'll check the case again and create a new BZ if needed. (In reply to Takayoshi Tanaka from comment #21) > The customer is facing the similar issue when installing CNS with OpenShift > 3.9. Could you tell me setting the following parameters can be worked around? > > openshift_storage_glusterfs_heketi_image=registry.access.redhat.com/rhgs3/ > rhgs-volmanager-rhel7 > openshift_storage_glusterfs_image=registry.access.redhat.com/rhgs3/rhgs- > server-rhel7 > openshift_storage_glusterfs_block_image=registry.access.redhat.com/rhgs3/ > rhgs-gluster-block-prov-rhel7 > openshift_storage_glusterfs_s3_image=registry.access.redhat.com/rhgs3/rhgs- > gluster-s3-server-rhel7 > > After setting these parameteres, the customer now faces the following error. > > exception: 'glusterfs_heketi_url' is undefined We have BZ #1583148 to track this issue in 3.9. For error "'glusterfs_heketi_url' is undefined", looks like another issue. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1816 |
Description of problem: Installation failed due to GlusterFS pods try to pull rhgs3/rhgs-server-rhel7:latest from docker.io. Version-Release number of the following components: atomic-openshift-3.10.0-0.50.0.git.0.db6dfd6.el7 openshift-ansible-3.10.0-0.50.0.git.0.bd68ade.el7 ansible-2.4.4.0-1.el7ae.noarch How reproducible: 100% Steps to Reproduce: 1. Install OCP with CNS 2. 3. Actual results: # ansible-playbook -i inventory -vv /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml ... TASK [openshift_storage_glusterfs : Wait for GlusterFS pods] ********************************************************************************************************************************* task path: /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/tasks/glusterfs_deploy.yml:102 Wednesday 23 May 2018 16:47:00 +0800 (0:00:02.212) 0:11:48.194 ********* FAILED - RETRYING: Wait for GlusterFS pods (30 retries left). ... FAILED - RETRYING: Wait for GlusterFS pods (1 retries left). fatal: [qe-weshi-cns-master-etcd-1.0523-wz1.qe.rhcloud.com]: FAILED! => {"attempts": 30, "changed": false, "failed": true, "results": {"cmd": "/usr/bin/oc get pod --selector=glusterfs=storage-pod -o json -n glusterfs", "results": [{"apiVersion": "v1", "items": [{"apiVersion": "v1", "kind": "Pod", "metadata": {"annotations": {"openshift.io/scc": "privileged"}, "creationTimestamp": "2018-05-23T08:47:07Z", "generateName": "glusterfs-storage-", "labels": {"controller-revision-hash": "2444950576", "glusterfs": "storage-pod", "glusterfs-node": "pod", "pod-template-generation": "1"}, "name": "glusterfs-storage-8hs2s", "namespace": "glusterfs", "ownerReferences": [{"apiVersion": "apps/v1", "blockOwnerDeletion": true, "controller": true, "kind": "DaemonSet", "name": "glusterfs-storage", "uid": "e0007487-5e65-11e8-bd1b-42010af00002"}], "resourceVersion": "2597", "selfLink": "/api/v1/namespaces/glusterfs/pods/glusterfs-storage-8hs2s", "uid": "e00a887e-5e65-11e8-bd1b-42010af00002"}, "spec": {"containers": [{"env": [{"name": "GB_GLFS_LRU_COUNT", "value": "15"}, {"name": "TCMU_LOGDIR", "value": "/var/log/glusterfs/gluster-block"}, {"name": "GB_LOGDIR", "value": "/var/log/glusterfs/gluster-block"}], "image": "rhgs3/rhgs-server-rhel7:latest", "imagePullPolicy": "IfNotPresent", "livenessProbe": {"exec": {"command": ["/bin/bash", "-c", "systemctl status glusterd.service"]}, "failureThreshold": 50, "initialDelaySeconds": 40, "periodSeconds": 25, "successThreshold": 1, "timeoutSeconds": 3}, "name": "glusterfs", "readinessProbe": {"exec": {"command": ["/bin/bash", "-c", "systemctl status glusterd.service"]}, "failureThreshold": 50, "initialDelaySeconds": 40, "periodSeconds": 25, "successThreshold": 1, "timeoutSeconds": 3}, "resources": {"requests": {"cpu": "100m", "memory": "100Mi"}}, "securityContext": {"capabilities": {}, "privileged": true}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/var/lib/heketi", "name": "glusterfs-heketi"}, {"mountPath": "/run", "name": "glusterfs-run"}, {"mountPath": "/run/lvm", "name": "glusterfs-lvm"}, {"mountPath": "/etc/glusterfs", "name": "glusterfs-etc"}, {"mountPath": "/var/log/glusterfs", "name": "glusterfs-logs"}, {"mountPath": "/var/lib/glusterd", "name": "glusterfs-config"}, {"mountPath": "/dev", "name": "glusterfs-dev"}, {"mountPath": "/var/lib/misc/glusterfsd", "name": "glusterfs-misc"}, {"mountPath": "/sys/fs/cgroup", "name": "glusterfs-cgroup", "readOnly": true}, {"mountPath": "/etc/ssl", "name": "glusterfs-ssl", "readOnly": true}, {"mountPath": "/usr/lib/modules", "name": "kernel-modules", "readOnly": true}, {"mountPath": "/etc/target", "name": "glusterfs-target"}, {"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount", "name": "default-token-9662s", "readOnly": true}]}], "dnsPolicy": "ClusterFirst", "hostNetwork": true, "imagePullSecrets": [{"name": "default-dockercfg-fgqb5"}], "nodeName": "qe-weshi-cns-glusterfs-node-1", "nodeSelector": {"glusterfs": "storage-host"}, "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "serviceAccount": "default", "serviceAccountName": "default", "terminationGracePeriodSeconds": 30, "tolerations": [{"effect": "NoSchedule", "key": "node.kubernetes.io/memory-pressure", "operator": "Exists"}, {"effect": "NoSchedule", "key": "node.kubernetes.io/disk-pressure", "operator": "Exists"}, {"effect": "NoExecute", "key": "node.kubernetes.io/not-ready", "operator": "Exists"}, {"effect": "NoExecute", "key": "node.kubernetes.io/unreachable", "operator": "Exists"}], "volumes": [{"hostPath": {"path": "/var/lib/heketi", "type": ""}, "name": "glusterfs-heketi"}, {"emptyDir": {}, "name": "glusterfs-run"}, {"hostPath": {"path": "/run/lvm", "type": ""}, "name": "glusterfs-lvm"}, {"hostPath": {"path": "/etc/glusterfs", "type": ""}, "name": "glusterfs-etc"}, {"hostPath": {"path": "/var/log/glusterfs", "type": ""}, "name": "glusterfs-logs"}, {"hostPath": {"path": "/var/lib/glusterd", "type": ""}, "name": "glusterfs-config"}, {"hostPath": {"path": "/dev", "type": ""}, "name": "glusterfs-dev"}, {"hostPath": {"path": "/var/lib/misc/glusterfsd", "type": ""}, "name": "glusterfs-misc"}, {"hostPath": {"path": "/sys/fs/cgroup", "type": ""}, "name": "glusterfs-cgroup"}, {"hostPath": {"path": "/etc/ssl", "type": ""}, "name": "glusterfs-ssl"}, {"hostPath": {"path": "/usr/lib/modules", "type": ""}, "name": "kernel-modules"}, {"hostPath": {"path": "/etc/target", "type": ""}, "name": "glusterfs-target"}, {"name": "default-token-9662s", "secret": {"defaultMode": 420, "secretName": "default-token-9662s"}}]}, "status": {"conditions": [{"lastProbeTime": null, "lastTransitionTime": "2018-05-23T08:47:08Z", "status": "True", "type": "Initialized"}, {"lastProbeTime": null, "lastTransitionTime": "2018-05-23T08:47:08Z", "message": "containers with unready status: [glusterfs]", "reason": "ContainersNotReady", "status": "False", "type": "Ready"}, {"lastProbeTime": null, "lastTransitionTime": "2018-05-23T08:47:08Z", "status": "True", "type": "PodScheduled"}], "containerStatuses": [{"image": "rhgs3/rhgs-server-rhel7:latest", "imageID": "", "lastState": {}, "name": "glusterfs", "ready": false, "restartCount": 0, "state": {"waiting": {"message": "Back-off pulling image \"rhgs3/rhgs-server-rhel7:latest\"", "reason": "ImagePullBackOff"}}}], "hostIP": "10.240.0.4", "phase": "Pending", "podIP": "10.240.0.4", "qosClass": "Burstable", "startTime": "2018-05-23T08:47:08Z"}}, {"apiVersion": "v1", "kind": "Pod", "metadata": {"annotations": {"openshift.io/scc": "privileged"}, "creationTimestamp": "2018-05-23T08:47:07Z", "generateName": "glusterfs-storage-", "labels": {"controller-revision-hash": "2444950576", "glusterfs": "storage-pod", "glusterfs-node": "pod", "pod-template-generation": "1"}, "name": "glusterfs-storage-9ls5f", "namespace": "glusterfs", "ownerReferences": [{"apiVersion": "apps/v1", "blockOwnerDeletion": true, "controller": true, "kind": "DaemonSet", "name": "glusterfs-storage", "uid": "e0007487-5e65-11e8-bd1b-42010af00002"}], "resourceVersion": "2589", "selfLink": "/api/v1/namespaces/glusterfs/pods/glusterfs-storage-9ls5f", "uid": "e00a2698-5e65-11e8-bd1b-42010af00002"}, "spec": {"containers": [{"env": [{"name": "GB_GLFS_LRU_COUNT", "value": "15"}, {"name": "TCMU_LOGDIR", "value": "/var/log/glusterfs/gluster-block"}, {"name": "GB_LOGDIR", "value": "/var/log/glusterfs/gluster-block"}], "image": "rhgs3/rhgs-server-rhel7:latest", "imagePullPolicy": "IfNotPresent", "livenessProbe": {"exec": {"command": ["/bin/bash", "-c", "systemctl status glusterd.service"]}, "failureThreshold": 50, "initialDelaySeconds": 40, "periodSeconds": 25, "successThreshold": 1, "timeoutSeconds": 3}, "name": "glusterfs", "readinessProbe": {"exec": {"command": ["/bin/bash", "-c", "systemctl status glusterd.service"]}, "failureThreshold": 50, "initialDelaySeconds": 40, "periodSeconds": 25, "successThreshold": 1, "timeoutSeconds": 3}, "resources": {"requests": {"cpu": "100m", "memory": "100Mi"}}, "securityContext": {"capabilities": {}, "privileged": true}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/var/lib/heketi", "name": "glusterfs-heketi"}, {"mountPath": "/run", "name": "glusterfs-run"}, {"mountPath": "/run/lvm", "name": "glusterfs-lvm"}, {"mountPath": "/etc/glusterfs", "name": "glusterfs-etc"}, {"mountPath": "/var/log/glusterfs", "name": "glusterfs-logs"}, {"mountPath": "/var/lib/glusterd", "name": "glusterfs-config"}, {"mountPath": "/dev", "name": "glusterfs-dev"}, {"mountPath": "/var/lib/misc/glusterfsd", "name": "glusterfs-misc"}, {"mountPath": "/sys/fs/cgroup", "name": "glusterfs-cgroup", "readOnly": true}, {"mountPath": "/etc/ssl", "name": "glusterfs-ssl", "readOnly": true}, {"mountPath": "/usr/lib/modules", "name": "kernel-modules", "readOnly": true}, {"mountPath": "/etc/target", "name": "glusterfs-target"}, {"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount", "name": "default-token-9662s", "readOnly": true}]}], "dnsPolicy": "ClusterFirst", "hostNetwork": true, "imagePullSecrets": [{"name": "default-dockercfg-fgqb5"}], "nodeName": "qe-weshi-cns-glusterfs-node-2", "nodeSelector": {"glusterfs": "storage-host"}, "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "serviceAccount": "default", "serviceAccountName": "default", "terminationGracePeriodSeconds": 30, "tolerations": [{"effect": "NoSchedule", "key": "node.kubernetes.io/memory-pressure", "operator": "Exists"}, {"effect": "NoExecute", "key": "node.kubernetes.io/not-ready", "operator": "Exists"}, {"effect": "NoExecute", "key": "node.kubernetes.io/unreachable", "operator": "Exists"}, {"effect": "NoSchedule", "key": "node.kubernetes.io/disk-pressure", "operator": "Exists"}], "volumes": [{"hostPath": {"path": "/var/lib/heketi", "type": ""}, "name": "glusterfs-heketi"}, {"emptyDir": {}, "name": "glusterfs-run"}, {"hostPath": {"path": "/run/lvm", "type": ""}, "name": "glusterfs-lvm"}, {"hostPath": {"path": "/etc/glusterfs", "type": ""}, "name": "glusterfs-etc"}, {"hostPath": {"path": "/var/log/glusterfs", "type": ""}, "name": "glusterfs-logs"}, {"hostPath": {"path": "/var/lib/glusterd", "type": ""}, "name": "glusterfs-config"}, {"hostPath": {"path": "/dev", "type": ""}, "name": "glusterfs-dev"}, {"hostPath": {"path": "/var/lib/misc/glusterfsd", "type": ""}, "name": "glusterfs-misc"}, {"hostPath": {"path": "/sys/fs/cgroup", "type": ""}, "name": "glusterfs-cgroup"}, {"hostPath": {"path": "/etc/ssl", "type": ""}, "name": "glusterfs-ssl"}, {"hostPath": {"path": "/usr/lib/modules", "type": ""}, "name": "kernel-modules"}, {"hostPath": {"path": "/etc/target", "type": ""}, "name": "glusterfs-target"}, {"name": "default-token-9662s", "secret": {"defaultMode": 420, "secretName": "default-token-9662s"}}]}, "status": {"conditions": [{"lastProbeTime": null, "lastTransitionTime": "2018-05-23T08:47:07Z", "status": "True", "type": "Initialized"}, {"lastProbeTime": null, "lastTransitionTime": "2018-05-23T08:47:07Z", "message": "containers with unready status: [glusterfs]", "reason": "ContainersNotReady", "status": "False", "type": "Ready"}, {"lastProbeTime": null, "lastTransitionTime": "2018-05-23T08:47:07Z", "status": "True", "type": "PodScheduled"}], "containerStatuses": [{"image": "rhgs3/rhgs-server-rhel7:latest", "imageID": "", "lastState": {}, "name": "glusterfs", "ready": false, "restartCount": 0, "state": {"waiting": {"message": "Back-off pulling image \"rhgs3/rhgs-server-rhel7:latest\"", "reason": "ImagePullBackOff"}}}], "hostIP": "10.240.0.5", "phase": "Pending", "podIP": "10.240.0.5", "qosClass": "Burstable", "startTime": "2018-05-23T08:47:07Z"}}, {"apiVersion": "v1", "kind": "Pod", "metadata": {"annotations": {"openshift.io/scc": "privileged"}, "creationTimestamp": "2018-05-23T08:47:07Z", "generateName": "glusterfs-storage-", "labels": {"controller-revision-hash": "2444950576", "glusterfs": "storage-pod", "glusterfs-node": "pod", "pod-template-generation": "1"}, "name": "glusterfs-storage-qnbrr", "namespace": "glusterfs", "ownerReferences": [{"apiVersion": "apps/v1", "blockOwnerDeletion": true, "controller": true, "kind": "DaemonSet", "name": "glusterfs-storage", "uid": "e0007487-5e65-11e8-bd1b-42010af00002"}], "resourceVersion": "2593", "selfLink": "/api/v1/namespaces/glusterfs/pods/glusterfs-storage-qnbrr", "uid": "e007793e-5e65-11e8-bd1b-42010af00002"}, "spec": {"containers": [{"env": [{"name": "GB_GLFS_LRU_COUNT", "value": "15"}, {"name": "TCMU_LOGDIR", "value": "/var/log/glusterfs/gluster-block"}, {"name": "GB_LOGDIR", "value": "/var/log/glusterfs/gluster-block"}], "image": "rhgs3/rhgs-server-rhel7:latest", "imagePullPolicy": "IfNotPresent", "livenessProbe": {"exec": {"command": ["/bin/bash", "-c", "systemctl status glusterd.service"]}, "failureThreshold": 50, "initialDelaySeconds": 40, "periodSeconds": 25, "successThreshold": 1, "timeoutSeconds": 3}, "name": "glusterfs", "readinessProbe": {"exec": {"command": ["/bin/bash", "-c", "systemctl status glusterd.service"]}, "failureThreshold": 50, "initialDelaySeconds": 40, "periodSeconds": 25, "successThreshold": 1, "timeoutSeconds": 3}, "resources": {"requests": {"cpu": "100m", "memory": "100Mi"}}, "securityContext": {"capabilities": {}, "privileged": true}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/var/lib/heketi", "name": "glusterfs-heketi"}, {"mountPath": "/run", "name": "glusterfs-run"}, {"mountPath": "/run/lvm", "name": "glusterfs-lvm"}, {"mountPath": "/etc/glusterfs", "name": "glusterfs-etc"}, {"mountPath": "/var/log/glusterfs", "name": "glusterfs-logs"}, {"mountPath": "/var/lib/glusterd", "name": "glusterfs-config"}, {"mountPath": "/dev", "name": "glusterfs-dev"}, {"mountPath": "/var/lib/misc/glusterfsd", "name": "glusterfs-misc"}, {"mountPath": "/sys/fs/cgroup", "name": "glusterfs-cgroup", "readOnly": true}, {"mountPath": "/etc/ssl", "name": "glusterfs-ssl", "readOnly": true}, {"mountPath": "/usr/lib/modules", "name": "kernel-modules", "readOnly": true}, {"mountPath": "/etc/target", "name": "glusterfs-target"}, {"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount", "name": "default-token-9662s", "readOnly": true}]}], "dnsPolicy": "ClusterFirst", "hostNetwork": true, "imagePullSecrets": [{"name": "default-dockercfg-fgqb5"}], "nodeName": "qe-weshi-cns-glusterfs-node-3", "nodeSelector": {"glusterfs": "storage-host"}, "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "serviceAccount": "default", "serviceAccountName": "default", "terminationGracePeriodSeconds": 30, "tolerations": [{"effect": "NoSchedule", "key": "node.kubernetes.io/memory-pressure", "operator": "Exists"}, {"effect": "NoExecute", "key": "node.kubernetes.io/not-ready", "operator": "Exists"}, {"effect": "NoExecute", "key": "node.kubernetes.io/unreachable", "operator": "Exists"}, {"effect": "NoSchedule", "key": "node.kubernetes.io/disk-pressure", "operator": "Exists"}], "volumes": [{"hostPath": {"path": "/var/lib/heketi", "type": ""}, "name": "glusterfs-heketi"}, {"emptyDir": {}, "name": "glusterfs-run"}, {"hostPath": {"path": "/run/lvm", "type": ""}, "name": "glusterfs-lvm"}, {"hostPath": {"path": "/etc/glusterfs", "type": ""}, "name": "glusterfs-etc"}, {"hostPath": {"path": "/var/log/glusterfs", "type": ""}, "name": "glusterfs-logs"}, {"hostPath": {"path": "/var/lib/glusterd", "type": ""}, "name": "glusterfs-config"}, {"hostPath": {"path": "/dev", "type": ""}, "name": "glusterfs-dev"}, {"hostPath": {"path": "/var/lib/misc/glusterfsd", "type": ""}, "name": "glusterfs-misc"}, {"hostPath": {"path": "/sys/fs/cgroup", "type": ""}, "name": "glusterfs-cgroup"}, {"hostPath": {"path": "/etc/ssl", "type": ""}, "name": "glusterfs-ssl"}, {"hostPath": {"path": "/usr/lib/modules", "type": ""}, "name": "kernel-modules"}, {"hostPath": {"path": "/etc/target", "type": ""}, "name": "glusterfs-target"}, {"name": "default-token-9662s", "secret": {"defaultMode": 420, "secretName": "default-token-9662s"}}]}, "status": {"conditions": [{"lastProbeTime": null, "lastTransitionTime": "2018-05-23T08:47:07Z", "status": "True", "type": "Initialized"}, {"lastProbeTime": null, "lastTransitionTime": "2018-05-23T08:47:07Z", "message": "containers with unready status: [glusterfs]", "reason": "ContainersNotReady", "status": "False", "type": "Ready"}, {"lastProbeTime": null, "lastTransitionTime": "2018-05-23T08:47:07Z", "status": "True", "type": "PodScheduled"}], "containerStatuses": [{"image": "rhgs3/rhgs-server-rhel7:latest", "imageID": "", "lastState": {}, "name": "glusterfs", "ready": false, "restartCount": 0, "state": {"waiting": {"message": "Back-off pulling image \"rhgs3/rhgs-server-rhel7:latest\"", "reason": "ImagePullBackOff"}}}], "hostIP": "10.240.0.6", "phase": "Pending", "podIP": "10.240.0.6", "qosClass": "Burstable", "startTime": "2018-05-23T08:47:07Z"}}], "kind": "List", "metadata": {"resourceVersion": "", "selfLink": ""}}], "returncode": 0}, "state": "list"} # rpm -q atomic-openshift atomic-openshift-3.10.0-0.50.0.git.0.db6dfd6.el7.x86_64 # oc describe po glusterfs-storage-8hs2s ... Image: rhgs3/rhgs-server-rhel7:latest Image ID: ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulling 5m kubelet, qe-weshi-cns-glusterfs-node-1 pulling image "rhgs3/rhgs-server-rhel7:latest" Warning Failed 5m kubelet, qe-weshi-cns-glusterfs-node-1 Failed to pull image "rhgs3/rhgs-server-rhel7:latest": rpc error: code = Unknown desc = repository docker.io/rhgs3/rhgs-server-rhel7 not found: does not exist or no pull access Warning Failed 5m kubelet, qe-weshi-cns-glusterfs-node-1 Error: ErrImagePull Normal BackOff 5m kubelet, qe-weshi-cns-glusterfs-node-1 Back-off pulling image "rhgs3/rhgs-server-rhel7:latest" Warning Failed 5m kubelet, qe-weshi-cns-glusterfs-node-1 Error: ImagePullBackOff Normal Pulling 3m (x4 over 4m) kubelet, qe-weshi-cns-glusterfs-node-1 pulling image "rhgs3/rhgs-server-rhel7:latest" Warning Failed 3m (x4 over 4m) kubelet, qe-weshi-cns-glusterfs-node-1 Failed to pull image "rhgs3/rhgs-server-rhel7:latest": rpc error: code = Unknown desc = repository docker.io/rhgs3/rhgs-server-rhel7 not found: does not exist or no pull access Warning Failed 3m (x4 over 4m) kubelet, qe-weshi-cns-glusterfs-node-1 Error: ErrImagePull Warning Failed 2m (x6 over 4m) kubelet, qe-weshi-cns-glusterfs-node-1 Error: ImagePullBackOff Normal BackOff 2m (x7 over 4m) kubelet, qe-weshi-cns-glusterfs-node-1 Back-off pulling image "rhgs3/rhgs-server-rhel7:latest" Expected results: GlusterFS pods shouldn't try to pull rhgs3/rhgs-server-rhel7:latest from docker.io. Additional info: It only appear in atomic-openshift-3.10.0-0.50.0.git.0.db6dfd6.el7, when deploy with atomic-openshift-3.10.0-0.47.0.git.0.2fffa04.el7, it pull image from registry.access.redhat.com: # rpm -q atomic-openshift atomic-openshift-3.10.0-0.47.0.git.0.2fffa04.el7 # oc describe po glusterfs-storage-2bwct -n glusterfs ... Image: rhgs3/rhgs-server-rhel7:latest Image ID: docker-pullable://registry.access.redhat.com/rhgs3/rhgs-server-rhel7@sha256:42dc831e5452cf3371fd16cfb96944a5d3d4a16e2f5833d69ee2805c92f3c8a2 ... Also, it works well in 3.9, looks like a regression.