Description of problem: cns-deployment tool fails to deploy CNS as the gluster pods were not up and it eventually aborted the setup. $$$$$$$$$$$$ # oc get pods NAME READY STATUS RESTARTS AGE glusterfs-b15tq 0/1 Running 0 5m glusterfs-bixbl 0/1 Running 1 5m glusterfs-f9dsr 0/1 Running 0 5m storage-project-router-1-z42i4 1/1 Running 0 10m $$$$$$$$$$$$ ########################### # cns-deploy topology.json --deploy-gluster --cli oc --templates_dir=/usr/share/heketi/templates --namespace storage-project --yes --log-file=/var/log/cns-deploy/1-latest-cns-deploy.log --verbose Using OpenShift CLI. NAME STATUS AGE storage-project Active 7m Using namespace "storage-project". template "deploy-heketi" created serviceaccount "heketi-service-account" created template "heketi" created template "glusterfs" created Marking 'dhcp47-1.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp47-1.lab.eng.blr.redhat.com" labeled Marking 'dhcp46-50.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp46-50.lab.eng.blr.redhat.com" labeled Marking 'dhcp47-136.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp47-136.lab.eng.blr.redhat.com" labeled Deploying GlusterFS pods. daemonset "glusterfs" created Waiting for GlusterFS pods to start ... Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 3s glusterfs-bixbl 0/1 ContainerCreating 0 3s glusterfs-f9dsr 0/1 ContainerCreating 0 3s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 5s glusterfs-bixbl 0/1 ContainerCreating 0 5s glusterfs-f9dsr 0/1 ContainerCreating 0 5s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 7s glusterfs-bixbl 0/1 ContainerCreating 0 7s glusterfs-f9dsr 0/1 ContainerCreating 0 7s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 9s glusterfs-bixbl 0/1 ContainerCreating 0 9s glusterfs-f9dsr 0/1 ContainerCreating 0 9s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 12s glusterfs-bixbl 0/1 ContainerCreating 0 12s glusterfs-f9dsr 0/1 ContainerCreating 0 12s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 14s glusterfs-bixbl 0/1 ContainerCreating 0 14s glusterfs-f9dsr 0/1 ContainerCreating 0 14s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 16s glusterfs-bixbl 0/1 ContainerCreating 0 16s glusterfs-f9dsr 0/1 ContainerCreating 0 16s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 18s glusterfs-bixbl 0/1 ContainerCreating 0 18s glusterfs-f9dsr 0/1 ContainerCreating 0 18s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 21s glusterfs-bixbl 0/1 ContainerCreating 0 21s glusterfs-f9dsr 0/1 ContainerCreating 0 21s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 23s glusterfs-bixbl 0/1 ContainerCreating 0 23s glusterfs-f9dsr 0/1 ContainerCreating 0 23s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 25s glusterfs-bixbl 0/1 ContainerCreating 0 25s glusterfs-f9dsr 0/1 ContainerCreating 0 25s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 27s glusterfs-bixbl 0/1 ContainerCreating 0 27s glusterfs-f9dsr 0/1 ContainerCreating 0 27s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 30s glusterfs-bixbl 0/1 ContainerCreating 0 30s glusterfs-f9dsr 0/1 ContainerCreating 0 30s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 32s glusterfs-bixbl 0/1 ContainerCreating 0 32s glusterfs-f9dsr 0/1 ContainerCreating 0 32s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 34s glusterfs-bixbl 0/1 ContainerCreating 0 34s glusterfs-f9dsr 0/1 ContainerCreating 0 34s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 36s glusterfs-bixbl 0/1 ContainerCreating 0 36s glusterfs-f9dsr 0/1 ContainerCreating 0 36s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 38s glusterfs-bixbl 0/1 ContainerCreating 0 38s glusterfs-f9dsr 0/1 ContainerCreating 0 38s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 41s glusterfs-bixbl 0/1 ContainerCreating 0 41s glusterfs-f9dsr 0/1 ContainerCreating 0 41s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 43s glusterfs-bixbl 0/1 ContainerCreating 0 43s glusterfs-f9dsr 0/1 ContainerCreating 0 43s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 45s glusterfs-bixbl 0/1 ContainerCreating 0 45s glusterfs-f9dsr 0/1 ContainerCreating 0 45s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 47s glusterfs-bixbl 0/1 ContainerCreating 0 47s glusterfs-f9dsr 0/1 ContainerCreating 0 47s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 50s glusterfs-bixbl 0/1 ContainerCreating 0 50s glusterfs-f9dsr 0/1 ContainerCreating 0 50s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 52s glusterfs-bixbl 0/1 ContainerCreating 0 52s glusterfs-f9dsr 0/1 ContainerCreating 0 52s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 54s glusterfs-bixbl 0/1 ContainerCreating 0 54s glusterfs-f9dsr 0/1 ContainerCreating 0 54s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 56s glusterfs-bixbl 0/1 ContainerCreating 0 56s glusterfs-f9dsr 0/1 ContainerCreating 0 56s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 59s glusterfs-bixbl 0/1 ContainerCreating 0 59s glusterfs-f9dsr 0/1 ContainerCreating 0 59s Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 1m glusterfs-bixbl 0/1 ContainerCreating 0 1m glusterfs-f9dsr 0/1 ContainerCreating 0 1m ..Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 1m glusterfs-bixbl 0/1 Running 0 1m glusterfs-f9dsr 0/1 ContainerCreating 0 1m .......................Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 ContainerCreating 0 2m glusterfs-bixbl 0/1 Running 0 2m glusterfs-f9dsr 0/1 ContainerCreating 0 2m .........................Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 Running 0 3m glusterfs-bixbl 0/1 Running 0 3m glusterfs-f9dsr 0/1 ContainerCreating 0 3m .......Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 Running 0 3m glusterfs-bixbl 0/1 Running 0 3m glusterfs-f9dsr 0/1 Running 0 3m ............Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 Running 0 3m glusterfs-bixbl 0/1 Running 1 3m glusterfs-f9dsr 0/1 Running 0 3m .....Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 Running 0 4m glusterfs-bixbl 0/1 Running 1 4m glusterfs-f9dsr 0/1 Running 0 4m ..........................Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 Running 0 5m glusterfs-bixbl 0/1 Running 1 5m glusterfs-f9dsr 0/1 Running 0 5m ...............Checking status of pods matching 'glusterfs-node=pod': glusterfs-b15tq 0/1 Running 1 5m glusterfs-bixbl 0/1 Running 1 5m glusterfs-f9dsr 0/1 Running 0 5m Timed out waiting for pods matching 'glusterfs-node=pod'. No resources found Error from server: deploymentconfig "heketi" not found Error from server: services "heketi" not found Error from server: routes "heketi" not found Error from server: services "heketi-storage-endpoints" not found serviceaccount "heketi-service-account" deleted template "deploy-heketi" deleted template "heketi" deleted Removing label from 'dhcp47-1.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp47-1.lab.eng.blr.redhat.com" labeled Removing label from 'dhcp46-50.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp46-50.lab.eng.blr.redhat.com" labeled Removing label from 'dhcp47-136.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp47-136.lab.eng.blr.redhat.com" labeled daemonset "glusterfs" deleted template "glusterfs" deleted ############################ Version-Release number of selected component (if applicable): heketi-client-3.1.0-14.el7rhgs.x86_64 cns-deploy-3.1.0-14.el7rhgs.x86_64 etcd-3.0.15-1.el7.x86_64 docker-common-1.12.5-12.el7.x86_64 docker-rhel-push-plugin-1.12.5-12.el7.x86_64 docker-1.12.5-12.el7.x86_64 docker-client-1.12.5-12.el7.x86_64 openshift-ansible-filter-plugins-3.4.44-1.git.0.efa61c6.el7.noarch openshift-ansible-playbooks-3.4.44-1.git.0.efa61c6.el7.noarch atomic-openshift-node-3.4.0.39-1.git.0.5f32f06.el7.x86_64 openshift-ansible-3.4.44-1.git.0.efa61c6.el7.noarch openshift-ansible-lookup-plugins-3.4.44-1.git.0.efa61c6.el7.noarch openshift-ansible-roles-3.4.44-1.git.0.efa61c6.el7.noarch atomic-openshift-utils-3.4.44-1.git.0.efa61c6.el7.noarch atomic-openshift-clients-3.4.0.39-1.git.0.5f32f06.el7.x86_64 openshift-ansible-docs-3.4.44-1.git.0.efa61c6.el7.noarch openshift-ansible-callback-plugins-3.4.44-1.git.0.efa61c6.el7.noarch atomic-openshift-3.4.0.39-1.git.0.5f32f06.el7.x86_64 atomic-openshift-master-3.4.0.39-1.git.0.5f32f06.el7.x86_64 tuned-profiles-atomic-openshift-node-3.4.0.39-1.git.0.5f32f06.el7.x86_64 atomic-openshift-sdn-ovs-3.4.0.39-1.git.0.5f32f06.el7.x86_64 How reproducible: Seen twice in 2 different setups while testing the latest RC build + RHEL 7.3.2 updates from Stage Steps to Reproduce: 1. Setup OCP 3.4 with the latest build from http://download.lab.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/3.4/latest/x86_64/os 2. Create and setup a router 3. Execute # cns-deploy topology.json --deploy-gluster --cli oc --templates_dir=/usr/share/heketi/templates --namespace storage-project --yes --log-file=/var/log/1-latest-cns-deploy.log --verbose Actual results: CNS deployment failed as glusterd failed to start in the pods Expected results: CNS deployment should be succesful. Additional info: ######################## # oc describe pod glusterfs-f9dsr Name: glusterfs-f9dsr Namespace: storage-project Security Policy: privileged Node: dhcp46-50.lab.eng.blr.redhat.com/10.70.46.50 Start Time: Wed, 11 Jan 2017 22:01:23 +0530 Labels: glusterfs-node=pod Status: Running IP: 10.70.46.50 Controllers: DaemonSet/glusterfs Containers: glusterfs: Container ID: docker://480bbb4912f6cd1e5073e32e5e9c553a0837643e867c5927ec87297730e93a0e Image: rhgs3/rhgs-server-rhel7:3.1.3-17 Image ID: docker-pullable://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhgs3/rhgs-server-rhel7@sha256:781ad1e4a9d229fa881702818b4e5b84022b9c0480fa210e5f6782a4605309a5 Port: State: Running Started: Wed, 11 Jan 2017 22:04:38 +0530 Ready: False Restart Count: 0 Liveness: exec [/bin/bash -c systemctl status glusterd.service] delay=100s timeout=3s period=10s #success=1 #failure=3 Readiness: exec [/bin/bash -c systemctl status glusterd.service] delay=100s timeout=3s period=10s #success=1 #failure=3 Volume Mounts: /dev from glusterfs-dev (rw) /etc/glusterfs from glusterfs-etc (rw) /run from glusterfs-run (rw) /run/lvm from glusterfs-lvm (rw) /sys/fs/cgroup from glusterfs-cgroup (ro) /var/lib/glusterd from glusterfs-config (rw) /var/lib/heketi from glusterfs-heketi (rw) /var/lib/misc/glusterfsd from glusterfs-misc (rw) /var/log/glusterfs from glusterfs-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-ibzaf (ro) Environment Variables: <none> Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: glusterfs-heketi: Type: HostPath (bare host directory volume) Path: /var/lib/heketi glusterfs-run: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: glusterfs-lvm: Type: HostPath (bare host directory volume) Path: /run/lvm glusterfs-etc: Type: HostPath (bare host directory volume) Path: /etc/glusterfs glusterfs-logs: Type: HostPath (bare host directory volume) Path: /var/log/glusterfs glusterfs-config: Type: HostPath (bare host directory volume) Path: /var/lib/glusterd glusterfs-dev: Type: HostPath (bare host directory volume) Path: /dev glusterfs-misc: Type: HostPath (bare host directory volume) Path: /var/lib/misc/glusterfsd glusterfs-cgroup: Type: HostPath (bare host directory volume) Path: /sys/fs/cgroup default-token-ibzaf: Type: Secret (a volume populated by a Secret) SecretName: default-token-ibzaf QoS Class: BestEffort Tolerations: <none> Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 5m 5m 1 {kubelet dhcp46-50.lab.eng.blr.redhat.com} spec.containers{glusterfs} Normal Pulling pulling image "rhgs3/rhgs-server-rhel7:3.1.3-17" 2m 2m 1 {kubelet dhcp46-50.lab.eng.blr.redhat.com} spec.containers{glusterfs} Normal Pulled Successfully pulled image "rhgs3/rhgs-server-rhel7:3.1.3-17" 1m 1m 1 {kubelet dhcp46-50.lab.eng.blr.redhat.com} spec.containers{glusterfs} Normal Created Created container with docker id 480bbb4912f6; Security:[seccomp=unconfined] 1m 1m 1 {kubelet dhcp46-50.lab.eng.blr.redhat.com} spec.containers{glusterfs} Normal Started Started container with docker id 480bbb4912f6 11s 11s 1 {kubelet dhcp46-50.lab.eng.blr.redhat.com} spec.containers{glusterfs} Warning Unhealthy Readiness probe failed: ● glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2017-01-11 16:34:39 UTC; 1min 43s ago Process: 113 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=255) Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com glusterd[113]: ERROR: failed to create logfile "/var/log/glusterfs/etc-glusterfs-glusterd.vol.log" (No such file or directory) Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com glusterd[113]: ERROR: failed to open logfile /var/log/glusterfs/etc-glusterfs-glusterd.vol.log Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: glusterd.service: control process exited, code=exited status=255 Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Failed to start GlusterFS, a clustered file-system server. Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state. Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed. 11s 11s 1 {kubelet dhcp46-50.lab.eng.blr.redhat.com} spec.containers{glusterfs} Warning Unhealthy Liveness probe failed: ● glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2017-01-11 16:34:39 UTC; 1min 43s ago Process: 113 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=255) Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com glusterd[113]: ERROR: failed to create logfile "/var/log/glusterfs/etc-glusterfs-glusterd.vol.log" (No such file or directory) Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com glusterd[113]: ERROR: failed to open logfile /var/log/glusterfs/etc-glusterfs-glusterd.vol.log Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: glusterd.service: control process exited, code=exited status=255 Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Failed to start GlusterFS, a clustered file-system server. Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state. Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed. 1s 1s 1 {kubelet dhcp46-50.lab.eng.blr.redhat.com} spec.containers{glusterfs} Warning Unhealthy Liveness probe failed: ● glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2017-01-11 16:34:39 UTC; 1min 53s ago Process: 113 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=255) Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com glusterd[113]: ERROR: failed to create logfile "/var/log/glusterfs/etc-glusterfs-glusterd.vol.log" (No such file or directory) Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com glusterd[113]: ERROR: failed to open logfile /var/log/glusterfs/etc-glusterfs-glusterd.vol.log Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: glusterd.service: control process exited, code=exited status=255 Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Failed to start GlusterFS, a clustered file-system server. Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state. Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed. 1s 1s 1 {kubelet dhcp46-50.lab.eng.blr.redhat.com} spec.containers{glusterfs} Warning Unhealthy Readiness probe failed: ● glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2017-01-11 16:34:39 UTC; 1min 53s ago Process: 113 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=255) Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com glusterd[113]: ERROR: failed to create logfile "/var/log/glusterfs/etc-glusterfs-glusterd.vol.log" (No such file or directory) Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com glusterd[113]: ERROR: failed to open logfile /var/log/glusterfs/etc-glusterfs-glusterd.vol.log Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: glusterd.service: control process exited, code=exited status=255 Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Failed to start GlusterFS, a clustered file-system server. Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state. Jan 11 16:34:39 dhcp46-50.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed. ########################
Looks to be a selinux issue, but I am not absolutely certain. Any luck if we disable selinux in the host ?
(In reply to Humble Chirammal from comment #3) > Looks to be a selinux issue, but I am not absolutely certain. Any luck if we > disable selinux in the host ? I've disabled SELinux in all the nodes and retested the same again. However, the issue still remains is what i noticed. ################################ [root@dhcp47-1 audit]# getenforce Permissive [root@dhcp47-1 ~]# sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 28 [root@dhcp46-50 audit]# getenforce Permissive [root@dhcp46-50 ~]# sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 28 [root@dhcp47-136 audit]# getenforce Permissive [root@dhcp47-136 ~]# sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 28 # cns-deploy topology.json --deploy-gluster --cli oc --templates_dir=/usr/share/heketi/templates --namespace storage-project --yes --log-file=/var/log/cns-deploy/5-latest-cns-deploy.log --verbose Using OpenShift CLI. NAME STATUS AGE storage-project Active 14h Using namespace "storage-project". template "deploy-heketi" created serviceaccount "heketi-service-account" created template "heketi" created template "glusterfs" created Marking 'dhcp47-1.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp47-1.lab.eng.blr.redhat.com" labeled Marking 'dhcp46-50.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp46-50.lab.eng.blr.redhat.com" labeled Marking 'dhcp47-136.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp47-136.lab.eng.blr.redhat.com" labeled Deploying GlusterFS pods. daemonset "glusterfs" created Waiting for GlusterFS pods to start ... Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 ContainerCreating 0 3s glusterfs-r58z0 0/1 ContainerCreating 0 3s glusterfs-vm5o6 0/1 ContainerCreating 0 3s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 ContainerCreating 0 5s glusterfs-r58z0 0/1 ContainerCreating 0 5s glusterfs-vm5o6 0/1 ContainerCreating 0 5s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 ContainerCreating 0 7s glusterfs-r58z0 0/1 ContainerCreating 0 7s glusterfs-vm5o6 0/1 ContainerCreating 0 7s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 9s glusterfs-r58z0 0/1 Running 0 9s glusterfs-vm5o6 0/1 Running 0 9s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 12s glusterfs-r58z0 0/1 Running 0 12s glusterfs-vm5o6 0/1 Running 0 12s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 14s glusterfs-r58z0 0/1 Running 0 14s glusterfs-vm5o6 0/1 Running 0 14s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 16s glusterfs-r58z0 0/1 Running 0 16s glusterfs-vm5o6 0/1 Running 0 16s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 18s glusterfs-r58z0 0/1 Running 0 18s glusterfs-vm5o6 0/1 Running 0 18s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 21s glusterfs-r58z0 0/1 Running 0 21s glusterfs-vm5o6 0/1 Running 0 21s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 23s glusterfs-r58z0 0/1 Running 0 23s glusterfs-vm5o6 0/1 Running 0 23s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 25s glusterfs-r58z0 0/1 Running 0 25s glusterfs-vm5o6 0/1 Running 0 25s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 27s glusterfs-r58z0 0/1 Running 0 27s glusterfs-vm5o6 0/1 Running 0 27s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 29s glusterfs-r58z0 0/1 Running 0 29s glusterfs-vm5o6 0/1 Running 0 29s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 32s glusterfs-r58z0 0/1 Running 0 32s glusterfs-vm5o6 0/1 Running 0 32s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 34s glusterfs-r58z0 0/1 Running 0 34s glusterfs-vm5o6 0/1 Running 0 34s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 36s glusterfs-r58z0 0/1 Running 0 36s glusterfs-vm5o6 0/1 Running 0 36s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 39s glusterfs-r58z0 0/1 Running 0 39s glusterfs-vm5o6 0/1 Running 0 39s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 41s glusterfs-r58z0 0/1 Running 0 41s glusterfs-vm5o6 0/1 Running 0 41s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 43s glusterfs-r58z0 0/1 Running 0 43s glusterfs-vm5o6 0/1 Running 0 43s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 45s glusterfs-r58z0 0/1 Running 0 45s glusterfs-vm5o6 0/1 Running 0 45s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 47s glusterfs-r58z0 0/1 Running 0 47s glusterfs-vm5o6 0/1 Running 0 47s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 50s glusterfs-r58z0 0/1 Running 0 50s glusterfs-vm5o6 0/1 Running 0 50s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 52s glusterfs-r58z0 0/1 Running 0 52s glusterfs-vm5o6 0/1 Running 0 52s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 54s glusterfs-r58z0 0/1 Running 0 54s glusterfs-vm5o6 0/1 Running 0 54s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 56s glusterfs-r58z0 0/1 Running 0 56s glusterfs-vm5o6 0/1 Running 0 56s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 59s glusterfs-r58z0 0/1 Running 0 59s glusterfs-vm5o6 0/1 Running 0 59s Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 1m glusterfs-r58z0 0/1 Running 0 1m glusterfs-vm5o6 0/1 Running 0 1m ..........................Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 2m glusterfs-r58z0 0/1 Running 0 2m glusterfs-vm5o6 0/1 Running 0 2m ..................Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 0 2m glusterfs-r58z0 0/1 Running 1 2m glusterfs-vm5o6 0/1 Running 0 2m Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 1 2m glusterfs-r58z0 0/1 Running 1 2m glusterfs-vm5o6 0/1 Running 1 2m ......Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 1 3m glusterfs-r58z0 0/1 Running 1 3m glusterfs-vm5o6 0/1 Running 1 3m ..........................Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 1 4m glusterfs-r58z0 0/1 Running 1 4m glusterfs-vm5o6 0/1 Running 1 4m .........................Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 1 5m glusterfs-r58z0 0/1 Running 1 5m glusterfs-vm5o6 0/1 Running 1 5m .Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 1 5m glusterfs-r58z0 0/1 Running 2 5m glusterfs-vm5o6 0/1 Running 2 5m Checking status of pods matching 'glusterfs-node=pod': glusterfs-8u6ce 0/1 Running 2 5m glusterfs-r58z0 0/1 Running 2 5m glusterfs-vm5o6 0/1 Running 2 5m .............Timed out waiting for pods matching 'glusterfs-node=pod'. No resources found Error from server: deploymentconfig "heketi" not found Error from server: services "heketi" not found Error from server: routes "heketi" not found Error from server: services "heketi-storage-endpoints" not found serviceaccount "heketi-service-account" deleted template "deploy-heketi" deleted template "heketi" deleted Removing label from 'dhcp47-1.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp47-1.lab.eng.blr.redhat.com" labeled Removing label from 'dhcp46-50.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp46-50.lab.eng.blr.redhat.com" labeled Removing label from 'dhcp47-136.lab.eng.blr.redhat.com' as a GlusterFS node. node "dhcp47-136.lab.eng.blr.redhat.com" labeled daemonset "glusterfs" deleted template "glusterfs" deleted ################################ I'll update the audit logs and other useful logs soon for further debugging.
[root@dhcp46-209 ~]# oc get pods NAME READY STATUS RESTARTS AGE glusterfs-8u6ce 0/1 Running 0 19s glusterfs-r58z0 0/1 Running 0 19s glusterfs-vm5o6 0/1 Running 0 19s storage-project-router-1-z42i4 1/1 Running 1 14h See the errors seen inside the gluster containers below: **************************************** [root@dhcp46-209 ~]# oc rsh glusterfs-8u6ce sh-4.2# systemctl status gluster-setup ● gluster-setup.service - Configuring GlusterFS in container Loaded: loaded (/etc/systemd/system/gluster-setup.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2017-01-12 06:27:20 UTC; 18s ago Process: 102 ExecStart=/usr/sbin/gluster-setup.sh (code=exited, status=1/FAILURE) Main PID: 102 (code=exited, status=1/FAILURE) Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: Starting Configuring GlusterFS in container... Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com gluster-setup.sh[102]: ls: cannot access /var/log/glusterfs: No such file or directory Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com gluster-setup.sh[102]: cp: cannot stat '/var/log/glusterfs_bkp/*': No such file or directory Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com gluster-setup.sh[102]: Failed to copy /var/log/glusterfs Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: gluster-setup.service: main process exited, code=exited, status=1/FAILURE Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: Failed to start Configuring GlusterFS in container. Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: Unit gluster-setup.service entered failed state. Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: gluster-setup.service failed. sh-4.2# systemctl status glusterd ● glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2017-01-12 06:27:20 UTC; 40s ago Process: 116 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=255) Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com glusterd[116]: ERROR: failed to create logfile "/var/log/glusterfs/etc-glusterfs-glusterd.vol.log" (No such file ...rectory) Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com glusterd[116]: ERROR: failed to open logfile /var/log/glusterfs/etc-glusterfs-glusterd.vol.log Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: glusterd.service: control process exited, code=exited status=255 Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: Failed to start GlusterFS, a clustered file-system server. Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state. Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed. Hint: Some lines were ellipsized, use -l to show in full. sh-4.2# systemctl status glusterd -l ● glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2017-01-12 06:27:20 UTC; 43s ago Process: 116 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=255) Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com glusterd[116]: ERROR: failed to create logfile "/var/log/glusterfs/etc-glusterfs-glusterd.vol.log" (No such file or directory) Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com glusterd[116]: ERROR: failed to open logfile /var/log/glusterfs/etc-glusterfs-glusterd.vol.log Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: glusterd.service: control process exited, code=exited status=255 Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: Failed to start GlusterFS, a clustered file-system server. Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state. Jan 12 06:27:20 dhcp47-1.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed. sh-4.2# systemctl status rpcbind ● rpcbind.service - RPC bind service Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled) Active: inactive (dead) ****************************************
# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination KUBE-NODEPORT-NON-LOCAL all -- anywhere anywhere /* Ensure that non-local NodePort traffic can flow */ KUBE-FIREWALL all -- anywhere anywhere ACCEPT all -- anywhere anywhere /* traffic from docker */ ACCEPT all -- anywhere anywhere /* traffic from SDN */ ACCEPT udp -- anywhere anywhere multiport dports 4789 /* 001 vxlan incoming */ ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT icmp -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh OS_FIREWALL_ALLOW all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- 10.128.0.0/14 anywhere ACCEPT all -- anywhere 10.128.0.0/14 DOCKER-ISOLATION all -- anywhere anywhere DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT) target prot opt source destination KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */ KUBE-FIREWALL all -- anywhere anywhere Chain DOCKER (1 references) target prot opt source destination Chain DOCKER-ISOLATION (1 references) target prot opt source destination RETURN all -- anywhere anywhere Chain KUBE-FIREWALL (2 references) target prot opt source destination DROP all -- anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000 Chain KUBE-NODEPORT-NON-LOCAL (1 references) target prot opt source destination Chain KUBE-SERVICES (1 references) target prot opt source destination Chain OS_FIREWALL_ALLOW (1 references) target prot opt source destination ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:10250 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:http ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:https ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:10255 ACCEPT udp -- anywhere anywhere state NEW udp dpt:10255 ACCEPT udp -- anywhere anywhere state NEW udp dpt:4789 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:24007 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:24008 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:EtherNet/IP-1 ACCEPT tcp -- anywhere anywhere state NEW multiport dports 49152:49664
As requested while debugging the issue, I'm providing the following additional details as well from the gluster pods: ************************************* [root@dhcp46-209 ~]# oc get pods NAME READY STATUS RESTARTS AGE glusterfs-3vcx8 0/1 Running 0 12s glusterfs-btc16 0/1 Running 0 12s glusterfs-nalx7 0/1 Running 0 12s storage-project-router-1-z42i4 1/1 Running 1 15h [root@dhcp46-209 ~]# oc rsh glusterfs-3vcx8 sh-4.2# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/dm-10 xfs 10G 292M 9.7G 3% / devtmpfs devtmpfs 24G 0 24G 0% /dev shm tmpfs 64M 0 64M 0% /dev/shm /dev/sdb1 xfs 40G 477M 40G 2% /run /dev/mapper/rhel_dhcp47--183-root xfs 50G 1.9G 49G 4% /etc/glusterfs tmpfs tmpfs 24G 26M 24G 1% /run/lvm tmpfs tmpfs 24G 0 24G 0% /sys/fs/cgroup tmpfs tmpfs 24G 16K 24G 1% /run/secrets/kubernetes.io/serviceaccount tmpfs tmpfs 4.0E 0 4.0E 0% /tmp tmpfs tmpfs 4.0E 4.0K 4.0E 1% /var/log sh-4.2# mount /dev/mapper/docker-8:17-33554517-e9ea3ec5b1b81940f6ec9a54390ce536b84a91273264b2f7d1ef90d69666e76f on / type xfs (rw,relatime,seclabel,nouuid,attr2,inode64,sunit=1024,swidth=1024,noquota) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel) devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=24637396k,nr_inodes=6159349,mode=755) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000) mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel) /dev/sdb1 on /run type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/sdb1 on /etc/resolv.conf type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/mapper/rhel_dhcp47--183-root on /etc/glusterfs type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/sdb1 on /dev/termination-log type xfs (rw,relatime,seclabel,attr2,inode64,noquota) shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,context="system_u:object_r:svirt_sandbox_file_t:s0:c474,c506",size=65536k) /dev/sdb1 on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/sdb1 on /etc/hostname type xfs (rw,relatime,seclabel,attr2,inode64,noquota) tmpfs on /run/lvm type tmpfs (rw,nosuid,nodev,seclabel,mode=755) /dev/sdb1 on /run/secrets type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/sdb1 on /var/lib/heketi type xfs (rw,relatime,seclabel,attr2,inode64,noquota) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) /dev/sdb1 on /var/lib/glusterd type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/sdb1 on /var/log/glusterfs type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/sdb1 on /var/lib/misc/glusterfsd type xfs (rw,relatime,seclabel,attr2,inode64,noquota) tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime,rootcontext=system_u:object_r:svirt_sandbox_file_t:s0,seclabel) tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=4503599627370496k) tmpfs on /var/log type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=4503599627370496k) /dev/sdb1 on /var/log/journal/3bd7ff1f60c92b19bb03ab05ec9aae08 type xfs (rw,relatime,seclabel,attr2,inode64,noquota) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct) configfs on /sys/kernel/config type configfs (rw,relatime) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) nfsd on /proc/fs/nfsd type nfsd (rw,relatime) sh-4.2# cd /etc/glusterfs sh-4.2# ls gluster-rsyslog-5.8.conf gluster-rsyslog-7.2.conf glusterd.vol glusterfs-georep-logrotate glusterfs-logrotate group-virt.example logger.conf.example sh-4.2# ls -al total 36 drwxr-xr-x. 2 root root 204 Jan 12 06:27 . drwxr-xr-x. 64 root root 4096 Jan 12 08:24 .. -rw-r--r--. 1 root root 1822 Jan 12 06:27 gluster-rsyslog-5.8.conf -rw-r--r--. 1 root root 2564 Jan 12 06:27 gluster-rsyslog-7.2.conf -rw-r--r--. 1 root root 358 Jan 12 06:27 glusterd.vol -rw-r--r--. 1 root root 1001 Jan 12 06:27 glusterfs-georep-logrotate -rw-r--r--. 1 root root 513 Jan 12 06:27 glusterfs-logrotate -rw-r--r--. 1 root root 220 Jan 12 06:27 group-virt.example -rw-r--r--. 1 root root 338 Jan 12 06:27 logger.conf.example sh-4.2# sh-4.2# sh-4.2# cd /var/lib/glusterd sh-4.2# ls -al total 0 drwxr-xr-x. 2 root root 6 Jan 12 06:27 . drwxr-xr-x. 20 root root 265 Jan 12 08:24 .. *************************************
(In reply to Prasanth from comment #7) > As requested while debugging the issue, I'm providing the following > additional details as well from the gluster pods: > > ************************************* > [root@dhcp46-209 ~]# oc get pods > NAME READY STATUS RESTARTS AGE > glusterfs-3vcx8 0/1 Running 0 12s > glusterfs-btc16 0/1 Running 0 12s > glusterfs-nalx7 0/1 Running 0 12s > storage-project-router-1-z42i4 1/1 Running 1 15h > > > [root@dhcp46-209 ~]# oc rsh glusterfs-3vcx8 > sh-4.2# df -Th > Filesystem Type Size Used Avail Use% Mounted on > /dev/dm-10 xfs 10G 292M 9.7G 3% / > devtmpfs devtmpfs 24G 0 24G 0% /dev > shm tmpfs 64M 0 64M 0% /dev/shm > /dev/sdb1 xfs 40G 477M 40G 2% /run > /dev/mapper/rhel_dhcp47--183-root xfs 50G 1.9G 49G 4% > /etc/glusterfs > tmpfs tmpfs 24G 26M 24G 1% /run/lvm > tmpfs tmpfs 24G 0 24G 0% > /sys/fs/cgroup > tmpfs tmpfs 24G 16K 24G 1% > /run/secrets/kubernetes.io/serviceaccount > tmpfs tmpfs 4.0E 0 4.0E 0% /tmp > tmpfs tmpfs 4.0E 4.0K 4.0E 1% /var/log > > Thanks Prasanth for recording the state, yes, as we debugged bind mount is missing. We will update the details soon here.
Hi, Thanks for the setup prasanth, this reduced lot of time to replicate the setup. The Bug was in the Docker build 12(docker-1.12.5-14.el7.x86_64). RCA: rtalur, humble and myself saw the bindmount from host to container was failing due to some issue in docker. With which even subscription manager entitlements were not working as expected. To be more blunt, Anything bind mounted to a container under /var directory was failing and log files were cleaned up by default mount of docker which is '/var/log'. since glusterd could not find /var/log/glusterfs which is parent directory for all the log files, glusterd failed with no such file or directory. We also verified that it is not related to systemd as we found the issue even with heketi container when we tried bind mounting some file on /var/lib/something. Even without privileged We were able to hit this issue. Before filling a bug against docker, prasanth told us about the build which was added to the errata today. docker-1.12.5-14.el7.x86_64 I would request to retest the same with the above build. --Ashiq
Hi, I do not see the bind mount issue on /var on the docker-1.12.5-14.el7.x86_64 build. I still see the '/var/log' issue which is after all the startup process the /var/log is bind mounted from some tmpfs which overwrites the /var/log of the container so we do not have /var/log/glusterfs. --Ashiq
The mounts inside the container was overshadowed by some actions from oci-systemd-hook. That said, the /var/ mounts were actually happened but later oci hook overshadow the entire /var path which cause the glusterd service to fail. We have opened https://bugzilla.redhat.com/show_bug.cgi?id=1412728 and tracking the issue on the same bz.
This bug would be fixed with oci-systemd-hook-0.1.4-9.git671c428.el7.x86_64.rpm which is available @ https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=12348885 . Can we please try this package and try to reproduce this issue ?
(In reply to Humble Chirammal from comment #17) > This bug would be fixed with > oci-systemd-hook-0.1.4-9.git671c428.el7.x86_64.rpm which is available @ > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=12348885 . > > Can we please try this package and try to reproduce this issue ? Thanks Humble. That's a good news! Sure, I'll take this build now and will try to reproduce this issue again in a clean setup. I'll provide my updates here once it's done.
(In reply to Prasanth from comment #18) > (In reply to Humble Chirammal from comment #17) > > This bug would be fixed with > > oci-systemd-hook-0.1.4-9.git671c428.el7.x86_64.rpm which is available @ > > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=12348885 . > > > > Can we please try this package and try to reproduce this issue ? > > Thanks Humble. That's a good news! > > Sure, I'll take this build now and will try to reproduce this issue again in > a clean setup. I'll provide my updates here once it's done. I've quickly tried to deploy CNS after updating to this build (oci-systemd-hook-0.1.4-9.git671c428.el7) in my existing setup and looks like the deployment had gone through successfully. So looks like the reported issue is indeed fixed in this build.
(In reply to Prasanth from comment #19) > > I've quickly tried to deploy CNS after updating to this build > (oci-systemd-hook-0.1.4-9.git671c428.el7) in my existing setup and looks > like the deployment had gone through successfully. So looks like the > reported issue is indeed fixed in this build. Thanks Prasanth for quick verification!
(In reply to Prasanth from comment #18) > (In reply to Humble Chirammal from comment #17) > > This bug would be fixed with > > oci-systemd-hook-0.1.4-9.git671c428.el7.x86_64.rpm which is available @ > > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=12348885 . > > > > Can we please try this package and try to reproduce this issue ? > > Thanks Humble. That's a good news! > > Sure, I'll take this build now and will try to reproduce this issue again in > a clean setup. I'll provide my updates here once it's done. Tried again in a new clean setup and the cns-deploy was successful in deploying the CNS cluster, which was failing earlier. See the output below: ############ [root@dhcp47-183 ~]# date Fri Jan 13 19:57:46 IST 2017 [root@dhcp47-183 ~]# rpm -qa |grep oci oci-register-machine-0-1.11.gitdd0daef.el7.x86_64 oci-systemd-hook-0.1.4-9.git671c428.el7.x86_64 [root@dhcp47-183 ~]# rpm -qa |grep docker docker-selinux-1.10.3-57.el7.x86_64 docker-client-1.12.5-14.el7.x86_64 docker-1.12.5-14.el7.x86_64 cockpit-docker-126-1.el7.x86_64 docker-common-1.12.5-14.el7.x86_64 docker-rhel-push-plugin-1.12.5-14.el7.x86_64 [root@dhcp47-183 ~]# rpm -qa |grep atomic-openshift atomic-openshift-clients-3.4.0.39-1.git.0.5f32f06.el7.x86_64 atomic-openshift-utils-3.4.44-1.git.0.efa61c6.el7.noarch atomic-openshift-3.4.0.39-1.git.0.5f32f06.el7.x86_64 atomic-openshift-master-3.4.0.39-1.git.0.5f32f06.el7.x86_64 tuned-profiles-atomic-openshift-node-3.4.0.39-1.git.0.5f32f06.el7.x86_64 atomic-openshift-node-3.4.0.39-1.git.0.5f32f06.el7.x86_64 atomic-openshift-sdn-ovs-3.4.0.39-1.git.0.5f32f06.el7.x86_64 [root@dhcp47-183 ~]# rpm -qa |grep selinux selinux-policy-targeted-3.13.1-102.el7_3.13.noarch docker-selinux-1.10.3-57.el7.x86_64 libselinux-utils-2.5-6.el7.x86_64 libselinux-python-2.5-6.el7.x86_64 selinux-policy-3.13.1-102.el7_3.13.noarch container-selinux-1.12.5-14.el7.x86_64 libselinux-2.5-6.el7.x86_64 # oc get all NAME REVISION DESIRED CURRENT TRIGGERED BY dc/heketi 1 1 1 config dc/storage-project-router 1 1 1 config NAME DESIRED CURRENT READY AGE rc/heketi-1 1 1 1 32m rc/storage-project-router-1 1 1 1 48m NAME HOST/PORT PATH SERVICES PORT TERMINATION routes/heketi heketi-storage-project.cloudapps.mystorage.com ... 1 more heketi <all> NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/heketi 172.30.92.133 <none> 8080/TCP 32m svc/heketi-storage-endpoints 172.30.139.214 <none> 1/TCP 33m svc/storage-project-router 172.30.100.247 <none> 80/TCP,443/TCP,1936/TCP 48m NAME READY STATUS RESTARTS AGE po/glusterfs-6dxca 1/1 Running 0 46m po/glusterfs-fempf 1/1 Running 0 46m po/glusterfs-moge8 1/1 Running 0 46m po/heketi-1-6xmt0 1/1 Running 0 32m po/storage-project-router-1-ctgj9 1/1 Running 0 48m [root@dhcp47-183 ~]# heketi-cli volume list Id:02248b8b4e4b5723765494eeb4984ac0 Cluster:e0fc406952249928d0972bd9ffb9811b Name:heketidbstorage [root@dhcp47-183 ~]# heketi-cli volume info 02248b8b4e4b5723765494eeb4984ac0 Name: heketidbstorage Size: 2 Volume Id: 02248b8b4e4b5723765494eeb4984ac0 Cluster Id: e0fc406952249928d0972bd9ffb9811b Mount: 10.70.46.178:heketidbstorage Mount Options: backup-volfile-servers=10.70.46.198,10.70.47.82 Durability Type: replicate Distributed+Replica: 3 #################
Based on Comment 21 and Comment 22, marking this BZ as Verified. PS: Will start with the rest of the dynamic provisioning testing and validation once the Stage is back online.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2017-0148.html